2024-09-16T22:55:53.750 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T22:55:53.775 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T22:55:53.888 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656 branch: wip-dis-testing description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: idryomov@gmail.com first_in_suite: false job_id: '7907656' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-dis-testing ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: none client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 owner: scheduled_dis@teuthology priority: 70 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 2961 sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 sleep_before_teardown: 0 subset: 66/100 suite: rbd suite_branch: wip-dis-testing suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6e8474a02f2b86353394076ab80c6f50ea28e361/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 targets: smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJburlteIPoXlMMAu7agtyMe9MjtZIG6fiph3BK2/YJ+1V0KJNOfhlvM6TVwnXFsZpYH57jLXZ37V4WGK/WVzw0= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErEuV/FPcG3SWS+1PR+Jy91XcYhrZ8Hz+THycDjFSZFEK1ONydI009YoWnGW62UjkZtF2YEhMN/AlQBnyoykfw= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_19:45:14 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T22:55:53.889 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6e8474a02f2b86353394076ab80c6f50ea28e361/qa; will attempt to use it 2024-09-16T22:55:53.890 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6e8474a02f2b86353394076ab80c6f50ea28e361/qa/tasks 2024-09-16T22:55:53.890 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T22:55:53.892 INFO:teuthology.task.internal:Checking packages... 2024-09-16T22:55:53.927 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6e8474a02f2b86353394076ab80c6f50ea28e361' 2024-09-16T22:55:53.927 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T22:55:53.927 INFO:teuthology.packaging:ref: None 2024-09-16T22:55:53.927 INFO:teuthology.packaging:tag: None 2024-09-16T22:55:53.927 INFO:teuthology.packaging:branch: wip-dis-testing 2024-09-16T22:55:53.927 INFO:teuthology.packaging:sha1: 6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T22:55:53.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-dis-testing 2024-09-16T22:55:54.083 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4883.g6e8474a0 2024-09-16T22:55:54.084 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T22:55:54.092 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T22:55:54.092 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T22:55:54.100 INFO:teuthology.task.internal:Saving configuration 2024-09-16T22:55:54.118 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T22:55:54.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T22:55:54.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 22:50:53.415571', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJburlteIPoXlMMAu7agtyMe9MjtZIG6fiph3BK2/YJ+1V0KJNOfhlvM6TVwnXFsZpYH57jLXZ37V4WGK/WVzw0='} 2024-09-16T22:55:54.170 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 22:50:53.417403', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErEuV/FPcG3SWS+1PR+Jy91XcYhrZ8Hz+THycDjFSZFEK1ONydI009YoWnGW62UjkZtF2YEhMN/AlQBnyoykfw='} 2024-09-16T22:55:54.170 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T22:55:54.179 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-16T22:55:54.179 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-16T22:55:54.179 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T22:55:54.265 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f34d54fbac0>, signals=[15]) 2024-09-16T22:55:54.265 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T22:55:54.271 INFO:teuthology.task.internal:Opening connections... 2024-09-16T22:55:54.271 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T22:55:54.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:55:54.354 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T22:55:54.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:55:54.430 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T22:55:54.443 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-09-16T22:55:54.462 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-09-16T22:55:54.462 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-09-16T22:55:54.521 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-09-16T22:55:54.521 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-09-16T22:55:54.521 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-09-16T22:55:54.522 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-09-16T22:55:54.523 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T22:55:54.523 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-09-16T22:55:54.523 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T22:55:54.523 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T22:55:54.523 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T22:55:54.524 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-09-16T22:55:54.546 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-09-16T22:55:54.566 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-09-16T22:55:54.566 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-09-16T22:55:54.624 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-09-16T22:55:54.624 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-09-16T22:55:54.624 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-09-16T22:55:54.624 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-09-16T22:55:54.624 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-16T22:55:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-16T22:55:54.625 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-09-16T22:55:54.645 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T22:55:54.662 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T22:55:54.688 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T22:55:54.688 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-09-16T22:55:54.691 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-09-16T22:55:54.708 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T22:55:54.723 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T22:55:54.724 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-09-16T22:55:54.749 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-09-16T22:55:54.854 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T22:55:54.916 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T22:55:54.933 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T22:55:54.933 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T22:55:54.934 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T22:55:54.934 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T22:55:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T22:55:54.951 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-16T22:55:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T22:55:54.953 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-16T22:55:55.009 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-503.el9.x86_64 2024-09-16T22:55:55.009 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-503.el9.x86_64 2024-09-16T22:55:55.009 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-09-16T22:55:55.010 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-503.el9.x86_64 2024-09-16T22:55:55.010 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-503.el9.x86_64 2024-09-16T22:55:55.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-09-16T22:55:57.489 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-16T22:56:00.247 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.3 MB 00:03 2024-09-16T22:56:03.182 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-09-16T22:56:09.318 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-16T22:56:10.817 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 19 kB 00:00 2024-09-16T22:56:11.579 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-16T22:56:12.526 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 1.8 MB/s | 20 MB 00:11 2024-09-16T22:56:17.002 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-16T22:56:17.985 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-16T22:56:18.867 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T22:56:18.867 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T22:56:18.947 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T22:56:18.957 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T22:56:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-16T22:56:18.957 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T22:56:18.957 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T22:56:18.957 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-16T22:56:18.958 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:56:18.959 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 79 M 2024-09-16T22:56:18.959 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 126 M 2024-09-16T22:56:18.959 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T22:56:19.268 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.269 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.269 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.361 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.367 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.374 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.470 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 139 kB/s | 36 kB 00:00 2024-09-16T22:56:19.477 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-16T22:56:19.497 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:20.689 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-16T22:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 18 MB/s | 35 MB 00:01 2024-09-16T22:56:25.137 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.6 MB/s | 16 MB 00:05 2024-09-16T22:56:26.120 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-16T22:56:28.043 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-16T22:56:28.044 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-16T22:56:28.123 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T22:56:28.135 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T22:56:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-16T22:56:28.135 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T22:56:28.135 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T22:56:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-16T22:56:28.136 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T22:56:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-16T22:56:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-16T22:56:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-16T22:56:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 79 M 2024-09-16T22:56:28.137 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 126 M 2024-09-16T22:56:28.138 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T22:56:28.444 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.446 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.446 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-modules-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.544 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.550 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.557 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-modules-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.621 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 155 kB/s | 36 kB 00:00 2024-09-16T22:56:28.649 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-modules-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:28.676 INFO:teuthology.orchestra.run.smithi096.stdout:[MIRROR] kernel-modules-core-5.14.0-508.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-508.el9.x86_64.rpm (IP: 163.237.219.7) 2024-09-16T22:56:29.496 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 33 MB/s | 29 MB 00:00 2024-09-16T22:56:29.821 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T22:56:30.246 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 19 MB/s | 35 MB 00:01 2024-09-16T22:56:30.249 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T22:56:30.250 INFO:teuthology.orchestra.run.smithi096.stdout:Total 37 MB/s | 79 MB 00:02 2024-09-16T22:56:30.686 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.5 MB/s | 29 MB 00:11 2024-09-16T22:56:30.690 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T22:56:30.690 INFO:teuthology.orchestra.run.smithi098.stdout:Total 6.7 MB/s | 79 MB 00:11 2024-09-16T22:56:30.925 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T22:56:31.262 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T22:56:31.262 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T22:56:31.372 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T22:56:31.716 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T22:56:31.717 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T22:56:32.225 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T22:56:32.225 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T22:56:32.688 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T22:56:32.689 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T22:56:34.659 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T22:56:35.133 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T22:56:35.359 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-16T22:56:35.440 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:56:35.849 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-16T22:56:35.939 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:56:38.176 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:56:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:56:38.689 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:56:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:56:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:56:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:56:43.650 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:56:43.751 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:56:47.991 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:56:48.607 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.462 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.508 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.832 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.871 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.920 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:26.920 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-16T22:57:26.920 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:57:26.920 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:57:27.247 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:27.247 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-16T22:57:27.247 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-16T22:57:27.247 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:57:27.768 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T22:57:28.118 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.121 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:57:28.122 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T22:57:28.201 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.202 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.202 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T22:57:28.202 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T22:57:28.202 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-16T22:57:28.202 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T22:57:28.202 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T22:57:28.261 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.261 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.263 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-09-16T22:57:28.295 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-09-16T22:57:28.295 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 82.el9 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8274670 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-09-16T22:57:28.296 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T22:57:28.297 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T22:57:28.298 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T22:57:28.298 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.298 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T22:57:28.500 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T22:57:28.576 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.576 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.576 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.576 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T22:57:28.576 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T22:57:28.577 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-16T22:57:28.577 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T22:57:28.577 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-09-16T22:57:28.636 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-16T22:57:28.636 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-09-16T22:57:28.668 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-09-16T22:57:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-09-16T22:57:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-09-16T22:57:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 82.el9 2024-09-16T22:57:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-09-16T22:57:28.669 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8274670 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-16T22:57:28.670 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-09-16T22:57:28.671 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T22:57:28.672 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-09-16T22:57:28.673 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-16T22:57:28.673 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-508.el9.x86_64 2024-09-16T22:57:28.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-16T22:57:29.031 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-09-16T22:57:29.299 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-09-16T22:57:30.415 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T22:57:30.447 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-09-16T22:57:30.450 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-16T22:57:30.466 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.9LScwZiqd7 2024-09-16T22:57:30.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9LScwZiqd7 2024-09-16T22:57:30.533 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.9LScwZiqd7 2024-09-16T22:57:30.613 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-16T22:57:30.642 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-09-16T22:57:30.644 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-16T22:57:30.660 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.CHZC0b0BVu 2024-09-16T22:57:30.660 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CHZC0b0BVu 2024-09-16T22:57:30.675 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.9LScwZiqd7 is 6KB 2024-09-16T22:57:30.723 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.9LScwZiqd7 2024-09-16T22:57:30.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.CHZC0b0BVu 2024-09-16T22:57:30.738 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T22:57:30.802 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T22:57:30.802 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T22:57:30.802 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T22:57:30.802 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-16T22:57:30.802 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-16T22:57:30.879 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.CHZC0b0BVu is 6KB 2024-09-16T22:57:30.929 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.CHZC0b0BVu 2024-09-16T22:57:30.943 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-09-16T22:57:30.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-09-16T22:57:31.022 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-16T22:57:31.023 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-16T22:57:31.023 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-16T22:57:31.023 INFO:teuthology.orchestra.run.smithi098.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-16T22:57:31.024 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-16T22:57:31.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-09-16T22:58:00.988 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-16T22:58:00.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T22:58:00.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:58:01.175 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-16T22:58:01.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T22:58:01.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:58:19.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-16T22:58:19.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-16T22:58:28.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T22:58:28.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:58:28.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T22:58:28.583 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:58:31.393 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-16T22:58:31.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-09-16T22:58:43.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T22:58:43.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:58:43.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T22:58:43.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:59:01.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-16T22:59:15.464 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-16T22:59:15.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T22:59:15.739 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-16T22:59:15.739 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-09-16T22:59:15.794 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-508.el9.x86_64 2024-09-16T22:59:15.795 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-16T22:59:15.795 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T22:59:15.795 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T22:59:16.795 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-16T22:59:16.796 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T22:59:16.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T22:59:16.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T22:59:16.936 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-09-16T22:59:16.951 DEBUG:teuthology.parallel:result is None 2024-09-16T22:59:17.284 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-16T22:59:17.667 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T22:59:17.667 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-16T22:59:17.667 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-16T22:59:17.681 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-508.el9.x86_64 2024-09-16T22:59:17.682 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-16T22:59:17.682 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T22:59:17.682 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-16T22:59:18.683 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-16T22:59:18.683 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T22:59:18.773 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-09-16T22:59:18.793 DEBUG:teuthology.parallel:result is None 2024-09-16T22:59:18.793 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T22:59:18.801 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T22:59:18.802 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T22:59:18.804 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T22:59:18.820 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T22:59:18.827 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T22:59:18.836 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T22:59:18.836 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T22:59:18.857 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T22:59:18.899 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T22:59:18.905 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T22:59:18.906 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T22:59:18.912 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T22:59:18.938 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T22:59:18.948 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T22:59:18.967 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T22:59:18.977 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T22:59:18.979 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T22:59:18.985 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T22:59:18.985 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T22:59:18.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T22:59:19.044 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T22:59:19.056 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T22:59:19.056 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T22:59:19.058 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T22:59:19.099 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T22:59:19.151 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T22:59:19.192 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T22:59:19.272 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T22:59:19.308 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T22:59:19.308 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T22:59:19.369 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T22:59:19.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T22:59:19.458 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T22:59:19.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T22:59:19.581 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T22:59:19.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T22:59:19.646 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-09-16T22:59:19.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-09-16T22:59:19.697 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T22:59:19.731 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T22:59:20.211 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T22:59:20.219 INFO:teuthology.task.internal:Starting timer... 2024-09-16T22:59:20.219 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T22:59:20.229 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T22:59:20.237 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-09-16T22:59:20.293 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-09-16T22:59:20.294 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-09-16T22:59:20.332 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-09-16T22:59:20.333 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T22:59:20.334 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-09-16T22:59:20.359 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-09-16T22:59:20.360 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-09-16T22:59:20.402 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-09-16T22:59:20.403 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-09-16T22:59:20.403 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T22:59:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T22:59:20.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T22:59:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T22:59:20.468 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T22:59:20.468 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-09-16T22:59:20.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-09-16T22:59:20.588 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T22:59:20.597 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T22:59:20.600 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T22:59:20.720 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T22:59:20.731 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T22:59:20.732 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi096.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T23:03:01.240 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-09-16T23:03:01.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T23:03:01.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T23:03:01.317 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-16T23:03:01.404 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-16T23:03:01.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T23:03:01.405 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T23:03:01.480 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-09-16T23:03:01.560 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-09-16T23:03:01.560 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T23:03:01.571 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T23:03:01.572 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T23:03:01.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T23:03:01.575 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T23:03:01.575 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T23:03:01.602 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T23:03:01.618 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T23:03:01.629 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-16T23:03:01.643 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-09-16T23:03:01.644 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-16T23:03:01.654 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-09-16T23:03:01.669 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-09-16T23:03:01.669 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T23:03:01.679 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-09-16T23:03:01.683 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T23:03:01.694 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-16T23:03:01.708 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-16T23:03:01.742 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-16T23:03:01.745 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T23:03:01.745 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-16T23:03:01.745 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.748 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.748 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.748 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.760 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.764 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:03:01.765 INFO:teuthology.run_tasks:Running task install... 2024-09-16T23:03:01.773 DEBUG:teuthology.task.install:project ceph 2024-09-16T23:03:01.773 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}} 2024-09-16T23:03:01.773 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'} 2024-09-16T23:03:01.773 INFO:teuthology.task.install:Using flavor: default 2024-09-16T23:03:01.779 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T23:03:01.779 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-16T23:03:01.779 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'tag': None, 'wait_for_package': False} 2024-09-16T23:03:01.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:03:01.782 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'tag': None, 'wait_for_package': False} 2024-09-16T23:03:01.782 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:03:01.934 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/ 2024-09-16T23:03:01.934 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.g6e8474a0 2024-09-16T23:03:01.938 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/ 2024-09-16T23:03:01.938 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4883.g6e8474a0 2024-09-16T23:03:02.057 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T23:03:02.058 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:03:02.058 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T23:03:02.081 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-dis-testing/6e8474a02f2b86353394076ab80c6f50ea28e361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-16T23:03:02.081 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:03:02.081 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-16T23:03:02.084 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T23:03:02.084 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6e8474a02f2b86353394076ab80c6f50ea28e361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T23:03:02.107 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2024-09-16T23:03:02.107 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6e8474a02f2b86353394076ab80c6f50ea28e361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-16T23:03:02.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T23:03:02.173 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-16T23:03:02.225 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T23:03:02.249 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-16T23:03:02.289 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-09-16T23:03:02.291 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-09-16T23:03:02.313 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-09-16T23:03:02.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-09-16T23:03:02.523 INFO:teuthology.orchestra.run.smithi096.stdout:45 files removed 2024-09-16T23:03:02.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-09-16T23:03:02.551 INFO:teuthology.orchestra.run.smithi098.stdout:45 files removed 2024-09-16T23:03:02.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-09-16T23:03:03.113 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 301 kB/s | 84 kB 00:00 2024-09-16T23:03:03.122 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 320 kB/s | 84 kB 00:00 2024-09-16T23:03:03.403 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-16T23:03:03.422 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-16T23:03:03.688 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-16T23:03:03.707 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-16T23:03:05.512 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.3 MB 00:01 2024-09-16T23:03:08.443 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.3 MB 00:04 2024-09-16T23:03:09.968 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-09-16T23:03:11.442 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-16T23:03:15.564 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-16T23:03:17.478 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 6.5 MB 00:02 2024-09-16T23:03:17.590 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 19 kB 00:00 2024-09-16T23:03:18.461 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-16T23:03:19.061 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 19 kB 00:00 2024-09-16T23:03:20.298 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-16T23:03:23.916 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-16T23:03:25.654 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-16T23:03:26.142 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:03:26.150 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 11 M 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.5 M 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.2 M 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 5.5 M 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 22 M 2024-09-16T23:03:26.151 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 25 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 755 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 508 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 5.5 M 2024-09-16T23:03:26.152 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 45 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 143 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 162 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 320 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 307 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 99 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T23:03:26.153 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:Install 31 Packages 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:26.154 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 62 M 2024-09-16T23:03:26.155 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:03:27.100 INFO:teuthology.orchestra.run.smithi096.stdout:(1/33): ceph-base-19.3.0-4883.g6e8474a0.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-16T23:03:27.159 INFO:teuthology.orchestra.run.smithi096.stdout:(2/33): ceph-selinux-19.3.0-4883.g6e8474a0.el9. 432 kB/s | 25 kB 00:00 2024-09-16T23:03:27.251 INFO:teuthology.orchestra.run.smithi096.stdout:(3/33): libcephfs2-19.3.0-4883.g6e8474a0.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-16T23:03:27.385 INFO:teuthology.orchestra.run.smithi096.stdout:(4/33): ceph-radosgw-19.3.0-4883.g6e8474a0.el9. 13 MB/s | 11 MB 00:00 2024-09-16T23:03:27.410 INFO:teuthology.orchestra.run.smithi096.stdout:(5/33): libradosstriper1-19.3.0-4883.g6e8474a0. 3.1 MB/s | 508 kB 00:00 2024-09-16T23:03:27.469 INFO:teuthology.orchestra.run.smithi096.stdout:(6/33): python3-ceph-argparse-19.3.0-4883.g6e84 772 kB/s | 45 kB 00:00 2024-09-16T23:03:27.552 INFO:teuthology.orchestra.run.smithi096.stdout:(7/33): python3-ceph-common-19.3.0-4883.g6e8474 1.7 MB/s | 143 kB 00:00 2024-09-16T23:03:27.669 INFO:teuthology.orchestra.run.smithi096.stdout:(8/33): librgw2-19.3.0-4883.g6e8474a0.el9.x86_6 19 MB/s | 5.5 MB 00:00 2024-09-16T23:03:27.703 INFO:teuthology.orchestra.run.smithi096.stdout:(9/33): python3-cephfs-19.3.0-4883.g6e8474a0.el 1.1 MB/s | 162 kB 00:00 2024-09-16T23:03:27.728 INFO:teuthology.orchestra.run.smithi096.stdout:(10/33): python3-rados-19.3.0-4883.g6e8474a0.el 5.3 MB/s | 320 kB 00:00 2024-09-16T23:03:27.753 INFO:teuthology.orchestra.run.smithi096.stdout:(11/33): python3-rbd-19.3.0-4883.g6e8474a0.el9. 6.0 MB/s | 307 kB 00:00 2024-09-16T23:03:27.787 INFO:teuthology.orchestra.run.smithi096.stdout:(12/33): python3-rgw-19.3.0-4883.g6e8474a0.el9. 1.6 MB/s | 99 kB 00:00 2024-09-16T23:03:27.880 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:03:27.892 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:27.892 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:03:27.892 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 11 M 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.5 M 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.2 M 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 5.5 M 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 22 M 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 25 k 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-16T23:03:27.893 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 755 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 508 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 5.5 M 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 45 k 2024-09-16T23:03:27.894 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 143 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 162 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 320 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 307 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 99 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:27.895 INFO:teuthology.orchestra.run.smithi098.stdout:Install 31 Packages 2024-09-16T23:03:27.896 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-09-16T23:03:27.896 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:27.896 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 62 M 2024-09-16T23:03:27.896 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:03:27.988 INFO:teuthology.orchestra.run.smithi096.stdout:(13/33): ceph-common-19.3.0-4883.g6e8474a0.el9. 15 MB/s | 22 MB 00:01 2024-09-16T23:03:28.038 INFO:teuthology.orchestra.run.smithi096.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 376 kB/s | 107 kB 00:00 2024-09-16T23:03:28.080 INFO:teuthology.orchestra.run.smithi096.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-16T23:03:28.122 INFO:teuthology.orchestra.run.smithi096.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 510 kB/s | 171 kB 00:00 2024-09-16T23:03:28.172 INFO:teuthology.orchestra.run.smithi096.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 870 kB/s | 160 kB 00:00 2024-09-16T23:03:28.222 INFO:teuthology.orchestra.run.smithi096.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T23:03:28.248 INFO:teuthology.orchestra.run.smithi096.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 562 kB/s | 42 kB 00:00 2024-09-16T23:03:28.281 INFO:teuthology.orchestra.run.smithi096.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-09-16T23:03:28.307 INFO:teuthology.orchestra.run.smithi096.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 610 kB/s | 36 kB 00:00 2024-09-16T23:03:28.332 INFO:teuthology.orchestra.run.smithi096.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-16T23:03:28.357 INFO:teuthology.orchestra.run.smithi096.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 626 kB/s | 47 kB 00:00 2024-09-16T23:03:28.407 INFO:teuthology.orchestra.run.smithi096.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-16T23:03:28.433 INFO:teuthology.orchestra.run.smithi096.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-16T23:03:28.458 INFO:teuthology.orchestra.run.smithi096.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 298 kB/s | 26 kB 00:00 2024-09-16T23:03:28.483 INFO:teuthology.orchestra.run.smithi096.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T23:03:28.517 INFO:teuthology.orchestra.run.smithi096.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T23:03:28.567 INFO:teuthology.orchestra.run.smithi096.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-16T23:03:28.617 INFO:teuthology.orchestra.run.smithi096.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-16T23:03:28.664 INFO:teuthology.orchestra.run.smithi098.stdout:(1/33): ceph-base-19.3.0-4883.g6e8474a0.el9.x86 10 MB/s | 5.5 MB 00:00 2024-09-16T23:03:28.693 INFO:teuthology.orchestra.run.smithi096.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T23:03:28.731 INFO:teuthology.orchestra.run.smithi098.stdout:(2/33): ceph-selinux-19.3.0-4883.g6e8474a0.el9. 377 kB/s | 25 kB 00:00 2024-09-16T23:03:28.839 INFO:teuthology.orchestra.run.smithi098.stdout:(3/33): libcephfs2-19.3.0-4883.g6e8474a0.el9.x8 6.8 MB/s | 755 kB 00:00 2024-09-16T23:03:28.948 INFO:teuthology.orchestra.run.smithi098.stdout:(4/33): libradosstriper1-19.3.0-4883.g6e8474a0. 4.6 MB/s | 508 kB 00:00 2024-09-16T23:03:29.018 INFO:teuthology.orchestra.run.smithi096.stdout:(32/33): librados2-19.3.0-4883.g6e8474a0.el9.x8 7.7 MB/s | 3.5 MB 00:00 2024-09-16T23:03:29.068 INFO:teuthology.orchestra.run.smithi096.stdout:(33/33): librbd1-19.3.0-4883.g6e8474a0.el9.x86_ 7.2 MB/s | 3.2 MB 00:00 2024-09-16T23:03:29.073 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:03:29.073 INFO:teuthology.orchestra.run.smithi096.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-16T23:03:29.073 INFO:teuthology.orchestra.run.smithi098.stdout:(5/33): ceph-radosgw-19.3.0-4883.g6e8474a0.el9. 12 MB/s | 11 MB 00:00 2024-09-16T23:03:29.140 INFO:teuthology.orchestra.run.smithi098.stdout:(6/33): python3-ceph-argparse-19.3.0-4883.g6e84 679 kB/s | 45 kB 00:00 2024-09-16T23:03:29.178 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:03:29.216 INFO:teuthology.orchestra.run.smithi098.stdout:(7/33): python3-ceph-common-19.3.0-4883.g6e8474 1.9 MB/s | 143 kB 00:00 2024-09-16T23:03:29.216 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:03:29.216 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:03:29.316 INFO:teuthology.orchestra.run.smithi098.stdout:(8/33): librgw2-19.3.0-4883.g6e8474a0.el9.x86_6 15 MB/s | 5.5 MB 00:00 2024-09-16T23:03:29.341 INFO:teuthology.orchestra.run.smithi098.stdout:(9/33): python3-cephfs-19.3.0-4883.g6e8474a0.el 1.3 MB/s | 162 kB 00:00 2024-09-16T23:03:29.374 INFO:teuthology.orchestra.run.smithi098.stdout:(10/33): python3-rados-19.3.0-4883.g6e8474a0.el 5.3 MB/s | 320 kB 00:00 2024-09-16T23:03:29.583 INFO:teuthology.orchestra.run.smithi098.stdout:(11/33): ceph-common-19.3.0-4883.g6e8474a0.el9. 15 MB/s | 22 MB 00:01 2024-09-16T23:03:29.608 INFO:teuthology.orchestra.run.smithi098.stdout:(12/33): python3-rgw-19.3.0-4883.g6e8474a0.el9. 425 kB/s | 99 kB 00:00 2024-09-16T23:03:29.634 INFO:teuthology.orchestra.run.smithi098.stdout:(13/33): python3-rbd-19.3.0-4883.g6e8474a0.el9. 1.0 MB/s | 307 kB 00:00 2024-09-16T23:03:29.726 INFO:teuthology.orchestra.run.smithi098.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 753 kB/s | 107 kB 00:00 2024-09-16T23:03:29.751 INFO:teuthology.orchestra.run.smithi098.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 1.2 MB/s | 171 kB 00:00 2024-09-16T23:03:29.768 INFO:teuthology.orchestra.run.smithi098.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-16T23:03:29.793 INFO:teuthology.orchestra.run.smithi098.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-16T23:03:29.827 INFO:teuthology.orchestra.run.smithi098.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-09-16T23:03:29.844 INFO:teuthology.orchestra.run.smithi098.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-09-16T23:03:29.869 INFO:teuthology.orchestra.run.smithi098.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-16T23:03:29.886 INFO:teuthology.orchestra.run.smithi098.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 847 kB/s | 36 kB 00:00 2024-09-16T23:03:29.911 INFO:teuthology.orchestra.run.smithi098.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-16T23:03:29.928 INFO:teuthology.orchestra.run.smithi098.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 802 kB/s | 47 kB 00:00 2024-09-16T23:03:29.932 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:03:29.933 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:03:29.978 INFO:teuthology.orchestra.run.smithi098.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 524 kB/s | 26 kB 00:00 2024-09-16T23:03:29.995 INFO:teuthology.orchestra.run.smithi098.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.8 MB/s | 49 kB 00:00 2024-09-16T23:03:30.021 INFO:teuthology.orchestra.run.smithi098.stdout:(26/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-16T23:03:30.046 INFO:teuthology.orchestra.run.smithi098.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-16T23:03:30.079 INFO:teuthology.orchestra.run.smithi098.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-16T23:03:30.121 INFO:teuthology.orchestra.run.smithi098.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-16T23:03:30.197 INFO:teuthology.orchestra.run.smithi098.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-16T23:03:30.289 INFO:teuthology.orchestra.run.smithi098.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-16T23:03:30.364 INFO:teuthology.orchestra.run.smithi098.stdout:(32/33): librados2-19.3.0-4883.g6e8474a0.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-16T23:03:30.422 INFO:teuthology.orchestra.run.smithi098.stdout:(33/33): librbd1-19.3.0-4883.g6e8474a0.el9.x86_ 14 MB/s | 3.2 MB 00:00 2024-09-16T23:03:30.437 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:03:30.437 INFO:teuthology.orchestra.run.smithi098.stdout:Total 24 MB/s | 62 MB 00:02 2024-09-16T23:03:30.536 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:03:30.572 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:03:30.573 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:03:31.254 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:03:31.254 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:03:31.499 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:03:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T23:03:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T23:03:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T23:03:31.688 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T23:03:31.713 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T23:03:31.899 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T23:03:31.951 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T23:03:32.291 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T23:03:32.400 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:32.476 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:32.577 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 8/35 2024-09-16T23:03:32.602 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 9/35 2024-09-16T23:03:32.665 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 9/35 2024-09-16T23:03:32.702 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T23:03:32.747 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9. 11/35 2024-09-16T23:03:32.817 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 12/35 2024-09-16T23:03:32.839 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:03:32.848 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 13/35 2024-09-16T23:03:32.902 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 13/35 2024-09-16T23:03:32.916 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-16T23:03:32.937 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T23:03:32.944 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-16T23:03:32.970 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-16T23:03:32.992 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T23:03:33.015 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-16T23:03:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T23:03:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T23:03:33.146 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T23:03:33.169 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T23:03:33.195 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T23:03:33.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-16T23:03:33.224 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T23:03:33.262 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T23:03:33.303 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T23:03:33.573 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-16T23:03:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T23:03:33.671 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 23/35 2024-09-16T23:03:33.688 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:33.754 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 23/35 2024-09-16T23:03:33.781 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:33.784 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 24/35 2024-09-16T23:03:33.820 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T23:03:33.871 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 8/35 2024-09-16T23:03:33.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 9/35 2024-09-16T23:03:33.944 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 9/35 2024-09-16T23:03:33.973 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-16T23:03:34.018 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9. 11/35 2024-09-16T23:03:34.088 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 12/35 2024-09-16T23:03:34.125 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 13/35 2024-09-16T23:03:34.173 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 13/35 2024-09-16T23:03:34.199 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-16T23:03:34.308 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T23:03:34.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-16T23:03:34.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 27/35 2024-09-16T23:03:34.350 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-16T23:03:34.412 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 27/35 2024-09-16T23:03:34.413 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-16T23:03:34.441 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-16T23:03:34.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-16T23:03:34.488 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 28/35 2024-09-16T23:03:34.503 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T23:03:34.525 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x8 29/35 2024-09-16T23:03:34.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-16T23:03:34.868 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-16T23:03:34.908 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 23/35 2024-09-16T23:03:34.991 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 23/35 2024-09-16T23:03:35.023 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 24/35 2024-09-16T23:03:35.065 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-16T23:03:35.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-16T23:03:35.594 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 27/35 2024-09-16T23:03:35.666 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 27/35 2024-09-16T23:03:35.751 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 28/35 2024-09-16T23:03:35.788 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x8 29/35 2024-09-16T23:03:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:37.485 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:37.525 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 31/35 2024-09-16T23:03:37.617 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 31/35 2024-09-16T23:03:37.617 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T23:03:37.617 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T23:03:37.617 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:37.660 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:38.095 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:38.134 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:38.703 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 30/35 2024-09-16T23:03:38.738 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 31/35 2024-09-16T23:03:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 31/35 2024-09-16T23:03:38.829 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-16T23:03:38.829 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-16T23:03:38.829 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:38.864 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:47.840 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:47.840 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-09-16T23:03:47.841 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:48.813 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/35 2024-09-16T23:03:48.838 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/35 2024-09-16T23:03:48.838 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:03:48.838 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T23:03:48.838 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T23:03:48.838 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T23:03:48.839 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:48.840 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T23:03:48.907 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T23:03:48.907 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-09-16T23:03:49.071 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-09-16T23:03:49.072 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:50.042 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/35 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/35 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-16T23:03:50.069 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:50.072 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T23:03:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-16T23:03:50.153 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/35 2024-09-16T23:03:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 5/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 6/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9. 8/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x8 9/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 10/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 11/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 12/35 2024-09-16T23:03:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 13/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T23:03:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T23:03:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/35 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.194 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T23:03:51.195 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:51.196 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:51.197 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:03:51.395 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:51.395 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 5/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_6 6/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 7/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9. 8/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x8 9/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 10/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 11/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 12/35 2024-09-16T23:03:51.396 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 13/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-16T23:03:51.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-16T23:03:51.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-16T23:03:51.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-16T23:03:51.398 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-16T23:03:51.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-16T23:03:51.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 32/35 2024-09-16T23:03:51.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-16T23:03:51.399 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/35 2024-09-16T23:03:51.566 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-09-16T23:03:52.153 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:29 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:03:52.291 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 50 M 2024-09-16T23:03:52.292 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 161 k 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-16T23:03:52.293 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:03:52.294 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 51 M 2024-09-16T23:03:52.294 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 217 M 2024-09-16T23:03:52.294 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:03:52.489 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-16T23:03:52.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:52.489 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.490 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.491 INFO:teuthology.orchestra.run.smithi098.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.492 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-20.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:52.493 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:03:52.557 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-16T23:03:52.599 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T23:03:52.666 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): libcephsqlite-19.3.0-4883.g6e8474a0.el9. 681 kB/s | 161 kB 00:00 2024-09-16T23:03:52.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-09-16T23:03:53.404 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:28 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:03:53.545 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 50 M 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 161 k 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-16T23:03:53.546 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 51 M 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 217 M 2024-09-16T23:03:53.547 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:03:53.820 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-16T23:03:53.862 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-16T23:03:53.904 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): libcephsqlite-19.3.0-4883.g6e8474a0.el9. 715 kB/s | 161 kB 00:00 2024-09-16T23:03:53.999 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): ceph-test-19.3.0-4883.g6e8474a0.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-16T23:03:54.002 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:03:54.002 INFO:teuthology.orchestra.run.smithi096.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-16T23:03:54.030 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:03:54.050 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:03:54.050 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:03:54.433 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:03:54.433 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:03:54.974 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:03:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T23:03:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T23:03:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/4 2024-09-16T23:03:55.254 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): ceph-test-19.3.0-4883.g6e8474a0.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-16T23:03:55.257 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:03:55.257 INFO:teuthology.orchestra.run.smithi098.stdout:Total 30 MB/s | 51 MB 00:01 2024-09-16T23:03:55.285 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:03:55.305 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:03:55.305 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:03:55.682 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:03:55.682 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:03:56.215 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:03:56.297 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-16T23:03:56.336 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-16T23:03:56.364 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/4 2024-09-16T23:03:59.868 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/4 2024-09-16T23:03:59.967 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/4 2024-09-16T23:04:01.179 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/4 2024-09-16T23:04:01.264 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/4 2024-09-16T23:04:02.523 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/4 2024-09-16T23:04:02.523 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/4 2024-09-16T23:04:02.523 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/4 2024-09-16T23:04:02.523 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T23:04:02.816 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:02.817 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:03.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-09-16T23:04:03.698 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:40 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/4 2024-09-16T23:04:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/4 2024-09-16T23:04:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/4 2024-09-16T23:04:03.773 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-16T23:04:03.839 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:03.842 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:03.842 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:03.842 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:03.842 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 6.6 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 2.4 M 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 1.5 M 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 291 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 4.8 M 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 17 M 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T23:04:03.843 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T23:04:03.844 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T23:04:03.845 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout:Install 37 Packages 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:03.846 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 31 M 2024-09-16T23:04:03.847 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 109 M 2024-09-16T23:04:03.847 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:04.065 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:04.217 INFO:teuthology.orchestra.run.smithi096.stdout:(1/37): ceph-19.3.0-4883.g6e8474a0.el9.x86_64.r 51 kB/s | 6.6 kB 00:00 2024-09-16T23:04:04.342 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-09-16T23:04:04.409 INFO:teuthology.orchestra.run.smithi096.stdout:(2/37): ceph-mds-19.3.0-4883.g6e8474a0.el9.x86_ 7.4 MB/s | 2.4 MB 00:00 2024-09-16T23:04:04.451 INFO:teuthology.orchestra.run.smithi096.stdout:(3/37): ceph-mgr-19.3.0-4883.g6e8474a0.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-09-16T23:04:04.552 INFO:teuthology.orchestra.run.smithi096.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4883.g6e84 2.8 MB/s | 291 kB 00:00 2024-09-16T23:04:04.677 INFO:teuthology.orchestra.run.smithi096.stdout:(5/37): ceph-mon-19.3.0-4883.g6e8474a0.el9.x86_ 10 MB/s | 4.8 MB 00:00 2024-09-16T23:04:04.744 INFO:teuthology.orchestra.run.smithi096.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-16T23:04:04.781 INFO:teuthology.orchestra.run.smithi096.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.9 MB/s | 106 kB 00:00 2024-09-16T23:04:04.823 INFO:teuthology.orchestra.run.smithi096.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T23:04:04.956 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:39 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:05.015 INFO:teuthology.orchestra.run.smithi096.stdout:(9/37): ceph-osd-19.3.0-4883.g6e8474a0.el9.x86_ 28 MB/s | 17 MB 00:00 2024-09-16T23:04:05.042 INFO:teuthology.orchestra.run.smithi096.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 574 kB/s | 125 kB 00:00 2024-09-16T23:04:05.082 INFO:teuthology.orchestra.run.smithi096.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-09-16T23:04:05.098 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:05.100 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 6.6 k 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 2.4 M 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 1.5 M 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 291 k 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 4.8 M 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 17 M 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-16T23:04:05.101 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-16T23:04:05.102 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-16T23:04:05.103 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout:Install 37 Packages 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:05.104 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 31 M 2024-09-16T23:04:05.105 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 109 M 2024-09-16T23:04:05.105 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:05.158 INFO:teuthology.orchestra.run.smithi096.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-16T23:04:05.250 INFO:teuthology.orchestra.run.smithi096.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 914 kB/s | 188 kB 00:00 2024-09-16T23:04:05.275 INFO:teuthology.orchestra.run.smithi096.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 297 kB/s | 35 kB 00:00 2024-09-16T23:04:05.300 INFO:teuthology.orchestra.run.smithi096.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 790 kB/s | 172 kB 00:00 2024-09-16T23:04:05.334 INFO:teuthology.orchestra.run.smithi096.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 916 kB/s | 77 kB 00:00 2024-09-16T23:04:05.359 INFO:teuthology.orchestra.run.smithi096.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-09-16T23:04:05.385 INFO:teuthology.orchestra.run.smithi096.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 465 kB/s | 51 kB 00:00 2024-09-16T23:04:05.427 INFO:teuthology.orchestra.run.smithi096.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-16T23:04:05.444 INFO:teuthology.orchestra.run.smithi096.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 743 kB/s | 43 kB 00:00 2024-09-16T23:04:05.469 INFO:teuthology.orchestra.run.smithi096.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.2 MB/s | 172 kB 00:00 2024-09-16T23:04:05.494 INFO:teuthology.orchestra.run.smithi096.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 139 kB/s | 22 kB 00:00 2024-09-16T23:04:05.519 INFO:teuthology.orchestra.run.smithi096.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-16T23:04:05.545 INFO:teuthology.orchestra.run.smithi096.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 996 kB/s | 23 kB 00:00 2024-09-16T23:04:05.570 INFO:teuthology.orchestra.run.smithi096.stdout:(25/37): python3-jaraco-classes-3.2.1-5.el9.noa 233 kB/s | 18 kB 00:00 2024-09-16T23:04:05.595 INFO:teuthology.orchestra.run.smithi096.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-16T23:04:05.620 INFO:teuthology.orchestra.run.smithi096.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-16T23:04:05.646 INFO:teuthology.orchestra.run.smithi096.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 920 kB/s | 46 kB 00:00 2024-09-16T23:04:05.671 INFO:teuthology.orchestra.run.smithi096.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-16T23:04:05.690 INFO:teuthology.orchestra.run.smithi098.stdout:(1/37): ceph-19.3.0-4883.g6e8474a0.el9.x86_64.r 54 kB/s | 6.6 kB 00:00 2024-09-16T23:04:05.697 INFO:teuthology.orchestra.run.smithi096.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-16T23:04:05.722 INFO:teuthology.orchestra.run.smithi096.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-16T23:04:05.747 INFO:teuthology.orchestra.run.smithi096.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-16T23:04:05.772 INFO:teuthology.orchestra.run.smithi096.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-16T23:04:05.798 INFO:teuthology.orchestra.run.smithi096.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-16T23:04:05.823 INFO:teuthology.orchestra.run.smithi096.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-16T23:04:05.848 INFO:teuthology.orchestra.run.smithi096.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-16T23:04:05.873 INFO:teuthology.orchestra.run.smithi096.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-16T23:04:05.877 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:05.877 INFO:teuthology.orchestra.run.smithi096.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-16T23:04:05.882 INFO:teuthology.orchestra.run.smithi098.stdout:(2/37): ceph-mgr-19.3.0-4883.g6e8474a0.el9.x86_ 4.9 MB/s | 1.5 MB 00:00 2024-09-16T23:04:05.932 INFO:teuthology.orchestra.run.smithi098.stdout:(3/37): ceph-mds-19.3.0-4883.g6e8474a0.el9.x86_ 6.6 MB/s | 2.4 MB 00:00 2024-09-16T23:04:06.019 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:06.032 INFO:teuthology.orchestra.run.smithi098.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4883.g6e84 2.8 MB/s | 291 kB 00:00 2024-09-16T23:04:06.059 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:06.059 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:06.108 INFO:teuthology.orchestra.run.smithi098.stdout:(5/37): ceph-mon-19.3.0-4883.g6e8474a0.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-16T23:04:06.208 INFO:teuthology.orchestra.run.smithi098.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-16T23:04:06.250 INFO:teuthology.orchestra.run.smithi098.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-16T23:04:06.292 INFO:teuthology.orchestra.run.smithi098.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-16T23:04:06.334 INFO:teuthology.orchestra.run.smithi098.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-16T23:04:06.501 INFO:teuthology.orchestra.run.smithi098.stdout:(10/37): ceph-osd-19.3.0-4883.g6e8474a0.el9.x86 28 MB/s | 17 MB 00:00 2024-09-16T23:04:06.526 INFO:teuthology.orchestra.run.smithi098.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-09-16T23:04:06.543 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:06.543 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:06.559 INFO:teuthology.orchestra.run.smithi098.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-16T23:04:06.580 INFO:teuthology.orchestra.run.smithi098.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.1 MB/s | 172 kB 00:00 2024-09-16T23:04:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 931 kB/s | 35 kB 00:00 2024-09-16T23:04:06.622 INFO:teuthology.orchestra.run.smithi098.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-16T23:04:06.639 INFO:teuthology.orchestra.run.smithi098.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-16T23:04:06.673 INFO:teuthology.orchestra.run.smithi098.stdout:(17/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-16T23:04:06.690 INFO:teuthology.orchestra.run.smithi098.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 619 kB/s | 42 kB 00:00 2024-09-16T23:04:06.732 INFO:teuthology.orchestra.run.smithi098.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-16T23:04:06.749 INFO:teuthology.orchestra.run.smithi098.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 204 kB/s | 22 kB 00:00 2024-09-16T23:04:06.766 INFO:teuthology.orchestra.run.smithi098.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 574 kB/s | 43 kB 00:00 2024-09-16T23:04:06.791 INFO:teuthology.orchestra.run.smithi098.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 612 kB/s | 11 kB 00:00 2024-09-16T23:04:06.816 INFO:teuthology.orchestra.run.smithi098.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-16T23:04:06.833 INFO:teuthology.orchestra.run.smithi098.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 440 kB/s | 18 kB 00:00 2024-09-16T23:04:06.850 INFO:teuthology.orchestra.run.smithi098.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 693 kB/s | 23 kB 00:00 2024-09-16T23:04:06.867 INFO:teuthology.orchestra.run.smithi098.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 581 kB/s | 19 kB 00:00 2024-09-16T23:04:06.884 INFO:teuthology.orchestra.run.smithi098.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-09-16T23:04:06.901 INFO:teuthology.orchestra.run.smithi098.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-16T23:04:06.926 INFO:teuthology.orchestra.run.smithi098.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-16T23:04:06.943 INFO:teuthology.orchestra.run.smithi098.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 996 kB/s | 16 kB 00:00 2024-09-16T23:04:06.968 INFO:teuthology.orchestra.run.smithi098.stdout:(31/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-16T23:04:06.994 INFO:teuthology.orchestra.run.smithi098.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-16T23:04:07.011 INFO:teuthology.orchestra.run.smithi098.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 852 kB/s | 36 kB 00:00 2024-09-16T23:04:07.036 INFO:teuthology.orchestra.run.smithi098.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-16T23:04:07.053 INFO:teuthology.orchestra.run.smithi098.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-16T23:04:07.078 INFO:teuthology.orchestra.run.smithi098.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-16T23:04:07.112 INFO:teuthology.orchestra.run.smithi098.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-16T23:04:07.116 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:07.116 INFO:teuthology.orchestra.run.smithi098.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-16T23:04:07.267 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:07.307 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:07.307 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:07.784 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:07.785 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:07.938 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:08.010 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T23:04:08.049 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T23:04:08.087 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T23:04:08.160 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T23:04:08.203 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T23:04:08.236 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T23:04:08.328 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T23:04:08.414 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T23:04:08.463 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T23:04:08.499 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T23:04:08.529 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T23:04:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T23:04:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T23:04:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T23:04:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T23:04:08.713 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T23:04:08.761 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T23:04:08.794 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T23:04:08.867 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T23:04:09.004 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T23:04:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T23:04:09.133 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T23:04:09.185 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T23:04:09.198 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-16T23:04:09.234 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-16T23:04:09.241 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T23:04:09.276 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-16T23:04:09.297 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T23:04:09.347 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-16T23:04:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-16T23:04:09.424 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-16T23:04:09.492 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T23:04:09.516 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-16T23:04:09.548 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T23:04:09.619 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-16T23:04:09.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T23:04:09.677 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-16T23:04:09.712 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-16T23:04:09.730 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T23:04:09.772 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T23:04:09.784 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-16T23:04:09.827 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-16T23:04:09.860 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-16T23:04:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T23:04:09.901 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-16T23:04:09.936 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-16T23:04:09.970 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-16T23:04:10.019 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-16T23:04:10.051 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-16T23:04:10.070 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9. 32/37 2024-09-16T23:04:10.099 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/37 2024-09-16T23:04:10.125 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/37 2024-09-16T23:04:10.125 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:10.125 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T23:04:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T23:04:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T23:04:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:10.137 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-16T23:04:10.276 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-16T23:04:10.364 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-16T23:04:10.417 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-16T23:04:10.456 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-16T23:04:10.504 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-16T23:04:10.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-16T23:04:10.764 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-16T23:04:10.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-16T23:04:10.896 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-16T23:04:10.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-16T23:04:11.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-16T23:04:11.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-16T23:04:11.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9. 32/37 2024-09-16T23:04:11.353 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/37 2024-09-16T23:04:11.379 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 33/37 2024-09-16T23:04:11.379 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:11.379 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-16T23:04:11.380 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T23:04:11.380 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-16T23:04:11.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:11.583 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/37 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/37 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T23:04:11.611 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:12.059 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 35/37 2024-09-16T23:04:12.083 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 35/37 2024-09-16T23:04:12.083 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:12.083 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T23:04:12.083 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T23:04:12.084 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T23:04:12.084 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:12.318 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 36/37 2024-09-16T23:04:12.341 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 36/37 2024-09-16T23:04:12.342 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:12.342 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T23:04:12.342 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T23:04:12.342 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T23:04:12.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:12.407 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 37/37 2024-09-16T23:04:12.829 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/37 2024-09-16T23:04:12.856 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 34/37 2024-09-16T23:04:12.856 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:12.856 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-16T23:04:12.856 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T23:04:12.857 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-16T23:04:12.857 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:13.304 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 35/37 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 35/37 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-16T23:04:13.328 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:13.556 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 36/37 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 36/37 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-16T23:04:13.579 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:13.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 37/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 37/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 5/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9. 6/37 2024-09-16T23:04:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T23:04:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T23:04:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T23:04:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T23:04:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T23:04:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T23:04:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T23:04:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T23:04:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T23:04:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T23:04:15.204 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T23:04:15.204 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:15.204 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:15.204 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:15.204 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T23:04:15.205 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T23:04:15.206 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:15.207 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 37/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 3/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 4/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 5/37 2024-09-16T23:04:15.528 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9. 6/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-16T23:04:15.529 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-16T23:04:15.530 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-16T23:04:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-16T23:04:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-16T23:04:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-16T23:04:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-16T23:04:15.531 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-16T23:04:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-16T23:04:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-16T23:04:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-16T23:04:15.532 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-16T23:04:15.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-09-16T23:04:16.204 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:16.293 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:04:16.335 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:16.336 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:04:16.336 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:16.410 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-09-16T23:04:16.499 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-16T23:04:16.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-16T23:04:16.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-16T23:04:16.501 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-16T23:04:16.502 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-16T23:04:16.503 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-16T23:04:16.503 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-16T23:04:16.503 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:16.503 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:16.906 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-09-16T23:04:16.981 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:17.111 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:17.112 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:17.112 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:17.112 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 784 k 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 784 k 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 791 k 2024-09-16T23:04:17.113 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:17.377 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-19.3.0-4883.g6e8474a0.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-16T23:04:17.377 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:17.378 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-16T23:04:17.378 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:17.383 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:17.383 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:17.389 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:17.389 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:17.463 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:17.506 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:52 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:17.595 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:04:17.637 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:17.638 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:04:17.638 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:17.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-09-16T23:04:17.931 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:18.015 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:18.272 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:53 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:18.341 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 784 k 2024-09-16T23:04:18.403 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 784 k 2024-09-16T23:04:18.404 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 791 k 2024-09-16T23:04:18.405 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:18.571 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:18.571 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:18.572 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:18.572 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:18.572 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:18.572 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:18.665 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.3.0-4883.g6e8474a0.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-16T23:04:18.665 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:18.665 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-16T23:04:18.666 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:18.671 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:18.671 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:18.677 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:18.677 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:18.760 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:18.788 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T23:04:19.218 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:19.327 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:19.369 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:56 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:19.506 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:19.506 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 145 k 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:19.507 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:04:19.508 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:19.508 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 145 k 2024-09-16T23:04:19.508 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 432 k 2024-09-16T23:04:19.508 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:19.716 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-19.3.0-4883.g6e8474 692 kB/s | 145 kB 00:00 2024-09-16T23:04:19.717 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:19.717 INFO:teuthology.orchestra.run.smithi096.stdout:Total 688 kB/s | 145 kB 00:00 2024-09-16T23:04:19.717 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:19.724 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:19.725 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:19.781 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:19.781 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:19.921 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:19.950 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:19.984 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:20.009 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:20.009 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:20.009 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T23:04:20.009 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:20.170 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-09-16T23:04:20.758 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:55 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:20.892 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 145 k 2024-09-16T23:04:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 432 k 2024-09-16T23:04:20.894 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:20.962 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:20.963 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:20.963 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:20.963 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:20.963 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:20.963 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:21.097 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.3.0-4883.g6e8474 715 kB/s | 145 kB 00:00 2024-09-16T23:04:21.097 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:21.098 INFO:teuthology.orchestra.run.smithi098.stdout:Total 709 kB/s | 145 kB 00:00 2024-09-16T23:04:21.098 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:21.105 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:21.106 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:21.163 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:21.163 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:21.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-09-16T23:04:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:21.431 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:21.431 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-16T23:04:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:21.771 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:21.860 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:04:21.902 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:21.903 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:04:21.904 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:21.980 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0. 1/1 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:22.381 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:22.551 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:59 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:22.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-09-16T23:04:22.681 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:22.682 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:22.682 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:22.682 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 14 M 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 34 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 16 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T23:04:22.683 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-16T23:04:22.684 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:22.685 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 19 M 2024-09-16T23:04:22.685 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 159 M 2024-09-16T23:04:22.685 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:23.188 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:23.277 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:04:23.304 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4883.g6e 239 kB/s | 34 kB 00:00 2024-09-16T23:04:23.320 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:23.321 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:04:23.321 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:23.329 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4883.g6e8 98 kB/s | 16 kB 00:00 2024-09-16T23:04:23.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-09-16T23:04:23.529 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-16T23:04:23.672 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-09-16T23:04:23.830 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4883.g6e8474a 21 MB/s | 14 MB 00:00 2024-09-16T23:04:23.872 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.7 MB/s | 548 kB 00:00 2024-09-16T23:04:23.897 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 300 kB/s | 19 kB 00:00 2024-09-16T23:04:23.931 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-16T23:04:23.945 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:58 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:23.965 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-09-16T23:04:23.990 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 527 kB/s | 31 kB 00:00 2024-09-16T23:04:24.040 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-16T23:04:24.073 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-09-16T23:04:24.074 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 14 M 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 34 k 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 16 k 2024-09-16T23:04:24.076 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-09-16T23:04:24.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:24.078 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:24.078 INFO:teuthology.orchestra.run.smithi096.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-16T23:04:24.078 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 19 M 2024-09-16T23:04:24.078 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 159 M 2024-09-16T23:04:24.079 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:24.126 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:24.145 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:24.146 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:24.438 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:24.439 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:25.051 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:25.051 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4883.g6e 228 kB/s | 34 kB 00:00 2024-09-16T23:04:25.068 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4883.g6e8 99 kB/s | 16 kB 00:00 2024-09-16T23:04:25.132 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T23:04:25.176 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T23:04:25.210 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-16T23:04:25.223 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T23:04:25.253 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T23:04:25.277 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-09-16T23:04:25.327 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-16T23:04:25.352 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 785 kB/s | 19 kB 00:00 2024-09-16T23:04:25.357 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T23:04:25.386 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-16T23:04:25.428 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-16T23:04:25.453 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-16T23:04:25.479 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-16T23:04:25.487 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T23:04:25.529 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-16T23:04:25.581 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T23:04:25.625 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T23:04:25.662 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T23:04:25.679 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-dashboard-19.3.0-4883.g6e8474 18 MB/s | 14 MB 00:00 2024-09-16T23:04:25.683 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:25.683 INFO:teuthology.orchestra.run.smithi098.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-16T23:04:25.702 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9 10/12 2024-09-16T23:04:25.732 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:25.751 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:25.751 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:26.031 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:26.031 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:26.624 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:26.696 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-16T23:04:26.740 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-16T23:04:26.787 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-16T23:04:26.817 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-16T23:04:26.923 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-16T23:04:27.061 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-16T23:04:27.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-16T23:04:27.239 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-16T23:04:27.267 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-16T23:04:27.308 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9 10/12 2024-09-16T23:04:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el 11/12 2024-09-16T23:04:28.907 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 12/12 2024-09-16T23:04:30.504 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el 11/12 2024-09-16T23:04:30.563 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 12/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 12/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el 1/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 2/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9 3/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T23:04:30.645 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T23:04:30.968 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T23:04:30.969 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T23:04:30.970 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T23:04:30.970 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T23:04:30.970 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T23:04:30.970 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:30.970 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:31.227 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T23:04:31.800 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:31.933 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 7.4 M 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T23:04:31.934 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout:======================================================================================= 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-16T23:04:31.935 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:31.936 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 38 M 2024-09-16T23:04:31.936 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 212 M 2024-09-16T23:04:31.936 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:32.481 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-16T23:04:32.523 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.4 MB/s | 801 kB 00:00 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 12/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el 1/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noa 2/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9 3/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-16T23:04:32.552 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-16T23:04:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-16T23:04:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-16T23:04:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-16T23:04:32.553 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-16T23:04:32.832 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 15 MB/s | 7.4 MB 00:00 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:32.862 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:32.863 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:33.090 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 54 kB/s | 33 kB 00:00 2024-09-16T23:04:33.105 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-16T23:04:33.315 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 38 kB/s | 18 kB 00:00 2024-09-16T23:04:33.674 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 66 kB/s | 38 kB 00:00 2024-09-16T23:04:33.685 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:08 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:33.757 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-09-16T23:04:33.818 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 7.4 M 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-16T23:04:33.819 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-09-16T23:04:33.820 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:33.821 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 38 M 2024-09-16T23:04:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 212 M 2024-09-16T23:04:33.825 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:34.154 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-16T23:04:34.196 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.4 MB/s | 801 kB 00:00 2024-09-16T23:04:34.438 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 16 MB/s | 7.4 MB 00:00 2024-09-16T23:04:34.449 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 316 kB/s | 245 kB 00:00 2024-09-16T23:04:34.463 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 106 kB/s | 33 kB 00:00 2024-09-16T23:04:34.525 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 4.1 MB/s | 4.9 MB 00:01 2024-09-16T23:04:34.663 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-16T23:04:34.722 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-16T23:04:34.908 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-09-16T23:04:35.189 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 525 kB/s | 245 kB 00:00 2024-09-16T23:04:35.225 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 323 kB/s | 250 kB 00:00 2024-09-16T23:04:35.347 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-16T23:04:35.465 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-16T23:04:35.690 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.8 MB/s | 4.9 MB 00:01 2024-09-16T23:04:35.983 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T23:04:35.985 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:35.985 INFO:teuthology.orchestra.run.smithi096.stdout:Total 9.4 MB/s | 38 MB 00:04 2024-09-16T23:04:36.342 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:36.369 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:36.369 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:36.373 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-09-16T23:04:36.804 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:36.804 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:36.882 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-16T23:04:36.882 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:36.883 INFO:teuthology.orchestra.run.smithi098.stdout:Total 12 MB/s | 38 MB 00:03 2024-09-16T23:04:37.237 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:37.264 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:37.264 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:37.578 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:37.645 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T23:04:37.694 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:37.694 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:37.698 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T23:04:37.797 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T23:04:37.833 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T23:04:38.017 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T23:04:38.048 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T23:04:38.125 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T23:04:38.452 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:38.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-16T23:04:38.580 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-16T23:04:38.678 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-16T23:04:38.714 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-16T23:04:38.794 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T23:04:38.897 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-16T23:04:38.912 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T23:04:38.930 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-16T23:04:39.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-16T23:04:39.668 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-16T23:04:39.785 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-16T23:04:40.167 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T23:04:41.025 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T23:04:41.184 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T23:04:41.207 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 12/12 2024-09-16T23:04:42.043 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-16T23:04:42.072 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 12/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 12/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 1/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T23:04:44.054 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T23:04:44.459 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T23:04:44.459 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T23:04:44.460 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T23:04:44.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T23:04:44.461 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:44.461 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 12/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e847 1/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-16T23:04:44.755 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-16T23:04:44.756 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-16T23:04:44.756 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-16T23:04:44.756 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-16T23:04:44.756 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-16T23:04:44.819 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-09-16T23:04:45.207 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-16T23:04:45.208 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-16T23:04:45.209 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-16T23:04:45.209 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-16T23:04:45.209 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:45.209 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:45.408 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:22 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:45.544 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 49 k 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T23:04:45.545 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-09-16T23:04:45.546 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:45.547 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.2 M 2024-09-16T23:04:45.547 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-09-16T23:04:45.547 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:45.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-09-16T23:04:45.767 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 26 kB 00:00 2024-09-16T23:04:45.792 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 19 kB 00:00 2024-09-16T23:04:45.818 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): ceph-mgr-rook-19.3.0-4883.g6e8474a0.el9 301 kB/s | 49 kB 00:00 2024-09-16T23:04:45.918 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-16T23:04:45.943 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-16T23:04:45.977 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-09-16T23:04:46.011 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-16T23:04:46.044 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 216 kB/s | 14 kB 00:00 2024-09-16T23:04:46.069 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.1 MB/s | 279 kB 00:00 2024-09-16T23:04:46.095 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-16T23:04:46.128 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.3 MB/s | 90 kB 00:00 2024-09-16T23:04:46.153 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 704 kB/s | 59 kB 00:00 2024-09-16T23:04:46.161 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:21 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:46.187 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-16T23:04:46.190 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:46.190 INFO:teuthology.orchestra.run.smithi096.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-16T23:04:46.253 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:46.268 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:46.269 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:46.293 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 49 k 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:46.294 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:46.295 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.2 M 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-09-16T23:04:46.296 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:46.357 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:46.357 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:46.862 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T23:04:47.041 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): ceph-mgr-rook-19.3.0-4883.g6e8474a0.el9 308 kB/s | 49 kB 00:00 2024-09-16T23:04:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T23:04:47.161 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T23:04:47.166 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 90 kB/s | 26 kB 00:00 2024-09-16T23:04:47.191 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 62 kB/s | 19 kB 00:00 2024-09-16T23:04:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T23:04:47.248 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T23:04:47.349 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T23:04:47.461 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T23:04:47.516 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T23:04:47.566 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 423 kB/s | 159 kB 00:00 2024-09-16T23:04:47.591 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 403 kB/s | 222 kB 00:00 2024-09-16T23:04:47.617 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 655 kB/s | 279 kB 00:00 2024-09-16T23:04:47.659 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-16T23:04:47.684 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 457 kB/s | 54 kB 00:00 2024-09-16T23:04:47.709 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-16T23:04:47.743 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-16T23:04:47.768 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-16T23:04:47.793 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-16T23:04:47.835 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-16T23:04:47.838 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:47.838 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-09-16T23:04:47.908 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:47.923 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:47.923 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:48.012 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:48.014 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:48.230 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T23:04:48.289 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T23:04:48.322 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T23:04:48.371 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T23:04:48.401 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 13/13 2024-09-16T23:04:48.528 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:48.707 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-16T23:04:48.760 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-16T23:04:48.816 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-16T23:04:48.846 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-16T23:04:48.887 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-16T23:04:48.988 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-16T23:04:49.089 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-16T23:04:49.139 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 13/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 1/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T23:04:49.153 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T23:04:49.154 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T23:04:49.490 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:49.491 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:49.751 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T23:04:49.848 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-16T23:04:49.903 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-16T23:04:49.936 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-16T23:04:49.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-16T23:04:50.037 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 13/13 2024-09-16T23:04:50.327 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:27 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:50.460 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 154 k 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T23:04:50.461 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 7.0 M 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-09-16T23:04:50.462 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:50.740 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-16T23:04:50.765 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4883.g6e8474a0.e 773 kB/s | 154 kB 00:00 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 13/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 1/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-16T23:04:50.781 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-16T23:04:50.782 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-16T23:04:50.782 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-16T23:04:50.782 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-16T23:04:50.782 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-16T23:04:50.824 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-16T23:04:50.850 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 688 kB/s | 58 kB 00:00 2024-09-16T23:04:50.874 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-16T23:04:51.118 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-16T23:04:51.119 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-16T23:04:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-16T23:04:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-16T23:04:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-16T23:04:51.120 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:51.120 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:51.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-09-16T23:04:51.954 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:26 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:52.083 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-16T23:04:52.086 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:52.086 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:52.086 INFO:teuthology.orchestra.run.smithi096.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 154 k 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-16T23:04:52.087 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 7.0 M 2024-09-16T23:04:52.088 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-09-16T23:04:52.089 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:52.157 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:52.169 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:52.169 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:52.269 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:52.270 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:52.496 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-16T23:04:52.547 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4883.g6e8474a0.e 605 kB/s | 154 kB 00:00 2024-09-16T23:04:52.577 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:52.597 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.3 MB/s | 548 kB 00:00 2024-09-16T23:04:52.614 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 860 kB/s | 58 kB 00:00 2024-09-16T23:04:52.640 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.7 MB/s | 74 kB 00:00 2024-09-16T23:04:52.760 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T23:04:52.806 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-16T23:04:52.811 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:52.811 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.7 MB/s | 7.0 MB 00:00 2024-09-16T23:04:52.827 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T23:04:52.880 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:52.891 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:52.891 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:52.991 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:52.991 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:53.303 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:53.350 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T23:04:53.439 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T23:04:53.471 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-16T23:04:53.507 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T23:04:53.524 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-16T23:04:53.531 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 6/6 2024-09-16T23:04:54.045 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-16T23:04:54.137 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-16T23:04:54.212 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-16T23:04:54.237 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 6/6 2024-09-16T23:04:54.318 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 6/6 2024-09-16T23:04:54.318 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/6 2024-09-16T23:04:54.318 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T23:04:54.318 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T23:04:54.319 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T23:04:54.319 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T23:04:54.598 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T23:04:54.599 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:54.599 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:54.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 6/6 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 1/6 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-16T23:04:55.063 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-16T23:04:55.342 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:55.343 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:55.460 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:32 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:55.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-09-16T23:04:55.593 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 899 k 2024-09-16T23:04:55.594 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-09-16T23:04:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 979 k 2024-09-16T23:04:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.9 M 2024-09-16T23:04:55.596 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:55.978 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): ceph-fuse-19.3.0-4883.g6e8474a0.el9.x86_ 3.1 MB/s | 899 kB 00:00 2024-09-16T23:04:56.161 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 172 kB/s | 80 kB 00:00 2024-09-16T23:04:56.162 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:56.162 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:31 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:56.162 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.7 MB/s | 979 kB 00:00 2024-09-16T23:04:56.173 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:56.195 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:56.195 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:56.279 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:56.279 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:56.297 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:56.298 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:56.298 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:56.299 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:56.299 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 899 k 2024-09-16T23:04:56.299 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-09-16T23:04:56.299 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-16T23:04:56.300 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:56.300 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:56.300 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:56.300 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-09-16T23:04:56.301 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:56.301 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 979 k 2024-09-16T23:04:56.301 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.9 M 2024-09-16T23:04:56.301 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:56.491 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:56.584 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 542 kB/s | 80 kB 00:00 2024-09-16T23:04:56.638 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T23:04:56.709 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.3.0-4883.g6e8474a0.el9.x86_ 3.2 MB/s | 899 kB 00:00 2024-09-16T23:04:56.709 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:56.710 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.3 MB/s | 979 kB 00:00 2024-09-16T23:04:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/2 2024-09-16T23:04:56.721 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:56.743 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:56.743 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:56.823 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:56.824 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:57.010 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:57.161 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-16T23:04:57.232 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/2 2024-09-16T23:04:57.429 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/2 2024-09-16T23:04:57.480 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/2 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:57.722 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:04:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2/2 2024-09-16T23:04:57.895 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/2 2024-09-16T23:04:58.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-volume 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:58.161 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:04:58.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-09-16T23:04:58.642 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:35 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:04:58.776 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 285 k 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 285 k 2024-09-16T23:04:58.777 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.3 M 2024-09-16T23:04:58.778 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:04:59.016 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:34 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:04:59.024 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-volume-19.3.0-4883.g6e8474a0.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-09-16T23:04:59.025 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:59.025 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-16T23:04:59.025 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:04:59.033 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:04:59.033 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:04:59.058 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:04:59.058 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:04:59.148 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.3.0-4883.g6e8474a0.el9 ceph-noarch 285 k 2024-09-16T23:04:59.149 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 285 k 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.3 M 2024-09-16T23:04:59.150 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:04:59.275 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:04:59.343 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-volume-19.3.0-4883.g6e8474a0.el9.noarch.rp 1.4 MB/s | 285 kB 00:00 2024-09-16T23:04:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:04:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-16T23:04:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:04:59.358 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:59.358 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:59.358 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T23:04:59.359 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:04:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:04:59.365 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:04:59.392 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:04:59.393 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:04:59.608 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:04:59.683 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:04:59.699 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:04:59.699 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-16T23:04:59.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:00.426 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:05:00.426 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:00.426 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:00.426 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:05:00.426 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:00.427 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 1/1 2024-09-16T23:05:00.716 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:00.717 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.3.0-4883.g6e8474a0.el9.noarch 2024-09-16T23:05:00.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:00.717 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:00.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-09-16T23:05:01.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-09-16T23:05:01.313 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:01.447 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 127 k 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:01.448 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 127 k 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-09-16T23:05:01.449 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:05:01.594 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:36 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:01.664 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-19.3.0-4883.g6e8474a0.el9.x86_64 590 kB/s | 127 kB 00:00 2024-09-16T23:05:01.665 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:01.665 INFO:teuthology.orchestra.run.smithi096.stdout:Total 585 kB/s | 127 kB 00:00 2024-09-16T23:05:01.665 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:05:01.671 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:05:01.671 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:05:01.727 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 127 k 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:05:01.729 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:01.730 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:05:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:01.730 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-09-16T23:05:01.730 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-09-16T23:05:01.730 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:05:01.733 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:05:01.734 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:05:01.882 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:05:01.936 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.3.0-4883.g6e8474a0.el9.x86_64 613 kB/s | 127 kB 00:00 2024-09-16T23:05:01.937 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:01.937 INFO:teuthology.orchestra.run.smithi098.stdout:Total 608 kB/s | 127 kB 00:00 2024-09-16T23:05:01.937 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:05:01.942 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:05:01.942 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:05:01.987 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:02.008 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:05:02.008 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:05:02.158 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:05:02.251 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:02.473 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:02.718 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:02.719 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:02.719 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:02.719 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:02.719 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:02.719 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:02.752 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:02.990 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-09-16T23:05:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:03.010 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:03.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:03.011 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:03.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-09-16T23:05:03.584 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:40 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:03.674 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:03.717 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:03.718 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:03.718 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:03.792 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-09-16T23:05:03.841 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:38 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:03.932 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:03.978 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:03.979 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:03.979 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:04.030 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-09-16T23:05:04.372 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:04.504 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 32 k 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:04.505 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:05:04.506 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:04.506 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 32 k 2024-09-16T23:05:04.506 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 142 k 2024-09-16T23:05:04.506 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:05:04.604 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:39 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:04.653 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-19.3.0-4883.g6e8474a0.el9.x86_6 216 kB/s | 32 kB 00:00 2024-09-16T23:05:04.653 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:04.653 INFO:teuthology.orchestra.run.smithi096.stdout:Total 214 kB/s | 32 kB 00:00 2024-09-16T23:05:04.654 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:05:04.657 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:05:04.657 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:05:04.671 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:05:04.672 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:05:04.735 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:04.735 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 32 k 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:05:04.736 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:04.737 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 32 k 2024-09-16T23:05:04.737 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 142 k 2024-09-16T23:05:04.737 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:05:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:05:04.875 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:04.882 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.3.0-4883.g6e8474a0.el9.x86_6 218 kB/s | 32 kB 00:00 2024-09-16T23:05:04.882 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:04.883 INFO:teuthology.orchestra.run.smithi098.stdout:Total 217 kB/s | 32 kB 00:00 2024-09-16T23:05:04.883 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:05:04.886 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:05:04.886 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:05:04.906 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:05:04.906 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:05:04.990 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:05:05.113 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:05.285 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:05.493 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:05.527 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:05.527 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:05.527 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:05.528 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:05.528 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:05.528 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:05.715 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:05.715 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:05.716 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:05.716 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:05.716 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:05.716 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:05.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-09-16T23:05:05.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-09-16T23:05:06.296 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:06.387 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:06.430 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:06.431 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:06.431 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:06.466 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:41 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:06.500 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-09-16T23:05:06.557 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:06.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-09-16T23:05:07.082 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:07.173 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:07.217 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:07.218 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:07.218 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:07.235 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:42 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:07.292 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-09-16T23:05:07.326 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:07.369 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:07.370 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:07.371 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:07.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-09-16T23:05:07.875 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:07.965 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:08.007 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:08.008 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:08.009 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:08.011 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:08.080 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-09-16T23:05:08.100 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:08.145 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:08.146 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:08.146 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:08.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-09-16T23:05:08.659 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:08.748 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:08.772 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:43 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:08.791 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:08.792 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:08.792 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:08.861 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:08.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-09-16T23:05:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:08.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-09-16T23:05:09.443 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:09.532 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:09.532 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:44 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:09.575 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:09.576 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:09.576 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:09.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-09-16T23:05:09.663 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:09.664 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:09.664 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:09.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-09-16T23:05:10.232 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:47 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:10.295 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:45 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:10.319 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:10.361 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:10.363 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:10.363 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:10.384 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:10.426 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:10.427 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:10.428 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:10.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-09-16T23:05:10.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-09-16T23:05:11.005 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:11.058 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:46 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:11.140 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:11.140 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:11.140 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-16T23:05:11.140 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:11.140 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 86 k 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 86 k 2024-09-16T23:05:11.141 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 231 k 2024-09-16T23:05:11.142 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:05:11.191 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 86 k 2024-09-16T23:05:11.192 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 86 k 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-09-16T23:05:11.193 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:05:11.331 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 452 kB/s | 86 kB 00:00 2024-09-16T23:05:11.331 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:11.332 INFO:teuthology.orchestra.run.smithi096.stdout:Total 449 kB/s | 86 kB 00:00 2024-09-16T23:05:11.332 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:05:11.339 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:05:11.339 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:05:11.385 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 446 kB/s | 86 kB 00:00 2024-09-16T23:05:11.386 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:11.386 INFO:teuthology.orchestra.run.smithi098.stdout:Total 443 kB/s | 86 kB 00:00 2024-09-16T23:05:11.386 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:05:11.393 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:05:11.394 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:05:11.403 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:05:11.403 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:05:11.457 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:05:11.457 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:05:11.550 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:05:11.602 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:05:11.653 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:11.699 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:12.014 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:12.050 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:12.286 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:12.286 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:12.286 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:12.286 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:12.287 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:12.287 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:12.299 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:12.501 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-09-16T23:05:12.559 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-09-16T23:05:13.118 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:50 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:13.157 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:48 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:05:13.262 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.2 M 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.2 M 2024-09-16T23:05:13.263 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 14 M 2024-09-16T23:05:13.264 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:05:13.299 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:13.299 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:13.299 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-09-16T23:05:13.299 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 3.2 M 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:13.300 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.2 M 2024-09-16T23:05:13.301 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 14 M 2024-09-16T23:05:13.301 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:05:13.644 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 8.4 MB/s | 3.2 MB 00:00 2024-09-16T23:05:13.644 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:13.645 INFO:teuthology.orchestra.run.smithi096.stdout:Total 8.4 MB/s | 3.2 MB 00:00 2024-09-16T23:05:13.645 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:05:13.649 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 9.2 MB/s | 3.2 MB 00:00 2024-09-16T23:05:13.649 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:13.650 INFO:teuthology.orchestra.run.smithi098.stdout:Total 9.1 MB/s | 3.2 MB 00:00 2024-09-16T23:05:13.650 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:05:13.654 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:05:13.654 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:05:13.659 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:05:13.660 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:05:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:05:13.737 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:05:13.745 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:05:13.745 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:05:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:05:14.208 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:05:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:14.274 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T23:05:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T23:05:14.301 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:15.366 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:15.366 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:15.367 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:15.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:15.367 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:15.467 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:15.467 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:15.467 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:15.468 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:15.468 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:15.468 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:15.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-09-16T23:05:15.734 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-09-16T23:05:16.194 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:51 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:16.323 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:16.328 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 170 k 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-09-16T23:05:16.329 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-09-16T23:05:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-09-16T23:05:16.330 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 170 k 2024-09-16T23:05:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 490 k 2024-09-16T23:05:16.330 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-09-16T23:05:16.458 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.3.0-4883.g6e8474a0.el9 ceph 170 k 2024-09-16T23:05:16.459 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 170 k 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 490 k 2024-09-16T23:05:16.460 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-16T23:05:16.527 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 865 kB/s | 170 kB 00:00 2024-09-16T23:05:16.528 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:16.529 INFO:teuthology.orchestra.run.smithi098.stdout:Total 858 kB/s | 170 kB 00:00 2024-09-16T23:05:16.529 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-09-16T23:05:16.536 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-09-16T23:05:16.536 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-09-16T23:05:16.600 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-09-16T23:05:16.600 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-09-16T23:05:16.692 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-19.3.0-4883.g6e8474a0.el9.x86_64.rpm 736 kB/s | 170 kB 00:00 2024-09-16T23:05:16.692 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-16T23:05:16.693 INFO:teuthology.orchestra.run.smithi096.stdout:Total 731 kB/s | 170 kB 00:00 2024-09-16T23:05:16.693 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-16T23:05:16.700 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-16T23:05:16.700 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-16T23:05:16.752 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-09-16T23:05:16.765 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-16T23:05:16.765 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-16T23:05:16.850 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:16.915 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-16T23:05:17.020 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:17.257 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:17.478 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:17.515 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:17.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 1/1 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.3.0-4883.g6e8474a0.el9.x86_64 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:17.741 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:18.000 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-09-16T23:05:18.318 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Sep 2024 11:03:25 PM UTC. 2024-09-16T23:05:18.410 INFO:teuthology.orchestra.run.smithi098.stdout:Package rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:18.455 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-09-16T23:05:18.457 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-09-16T23:05:18.457 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-09-16T23:05:18.515 DEBUG:teuthology.parallel:result is None 2024-09-16T23:05:18.598 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:55 ago on Mon 16 Sep 2024 11:03:23 PM UTC. 2024-09-16T23:05:18.690 INFO:teuthology.orchestra.run.smithi096.stdout:Package rbd-mirror-2:19.3.0-4883.g6e8474a0.el9.x86_64 is already installed. 2024-09-16T23:05:18.734 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-16T23:05:18.735 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-16T23:05:18.735 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-16T23:05:18.809 DEBUG:teuthology.parallel:result is None 2024-09-16T23:05:18.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:05:18.961 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T23:05:18.982 INFO:teuthology.orchestra.run.smithi096.stdout:19.3.0-4883.g6e8474a0.el9 2024-09-16T23:05:18.983 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.g6e8474a0.el9 2024-09-16T23:05:18.983 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.g6e8474a0 is installed. 2024-09-16T23:05:18.985 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:05:19.132 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-16T23:05:19.154 INFO:teuthology.orchestra.run.smithi098.stdout:19.3.0-4883.g6e8474a0.el9 2024-09-16T23:05:19.154 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4883.g6e8474a0.el9 2024-09-16T23:05:19.154 INFO:teuthology.task.install:The correct ceph version 19.3.0-4883.g6e8474a0 is installed. 2024-09-16T23:05:19.156 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T23:05:19.156 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:19.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T23:05:19.181 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:19.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T23:05:19.220 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T23:05:19.224 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:19.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T23:05:19.249 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T23:05:19.313 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:19.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T23:05:19.340 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T23:05:19.405 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T23:05:19.406 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:19.406 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T23:05:19.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T23:05:19.495 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:19.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T23:05:19.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T23:05:19.586 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T23:05:19.586 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:19.586 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T23:05:19.611 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T23:05:19.675 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:19.676 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T23:05:19.700 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T23:05:19.765 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T23:05:19.855 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T23:05:19.855 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-09-16T23:05:19.857 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-09-16T23:05:19.882 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T23:05:19.882 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T23:05:19.922 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T23:05:19.946 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T23:05:19.946 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T23:05:19.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T23:05:20.023 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-16T23:05:20.024 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T23:05:20.024 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7907656', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2961, 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'sleep_before_teardown': 0, 'subset': '66/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_6e8474a02f2b86353394076ab80c6f50ea28e361/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJburlteIPoXlMMAu7agtyMe9MjtZIG6fiph3BK2/YJ+1V0KJNOfhlvM6TVwnXFsZpYH57jLXZ37V4WGK/WVzw0=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErEuV/FPcG3SWS+1PR+Jy91XcYhrZ8Hz+THycDjFSZFEK1ONydI009YoWnGW62UjkZtF2YEhMN/AlQBnyoykfw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_19:45:14', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T23:05:20.025 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T23:05:20.051 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T23:05:20.067 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T23:05:20.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T23:05:20.131 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:20.131 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T23:05:20.185 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T23:05:20.185 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-09-16T23:05:20.240 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T23:05:20.240 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:20.240 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-16 23:05:14.533665323 +0000 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-16 23:02:41.120890958 +0000 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-16 23:02:41.120890958 +0000 2024-09-16T23:05:20.241 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-16 23:02:41.120890958 +0000 2024-09-16T23:05:20.241 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T23:05:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-16T23:05:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-16T23:05:20.304 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000203165 s, 2.5 MB/s 2024-09-16T23:05:20.305 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T23:05:20.361 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-09-16T23:05:20.416 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T23:05:20.416 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:20.416 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T23:05:20.416 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:20.416 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:20.417 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-16 23:05:14.534665300 +0000 2024-09-16T23:05:20.417 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-16 23:02:41.448883560 +0000 2024-09-16T23:05:20.417 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-16 23:02:41.448883560 +0000 2024-09-16T23:05:20.417 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-16 23:02:41.448883560 +0000 2024-09-16T23:05:20.417 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T23:05:20.479 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-16T23:05:20.479 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-16T23:05:20.479 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000205083 s, 2.5 MB/s 2024-09-16T23:05:20.481 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T23:05:20.537 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-09-16T23:05:20.592 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T23:05:20.592 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:20.592 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-16T23:05:20.592 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:20.592 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:20.593 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-16 23:05:14.534665300 +0000 2024-09-16T23:05:20.593 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-16 23:02:41.757876591 +0000 2024-09-16T23:05:20.593 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-16 23:02:41.757876591 +0000 2024-09-16T23:05:20.593 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-16 23:02:41.757876591 +0000 2024-09-16T23:05:20.593 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T23:05:20.655 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-16T23:05:20.655 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-16T23:05:20.655 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000206685 s, 2.5 MB/s 2024-09-16T23:05:20.656 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T23:05:20.712 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-09-16T23:05:20.767 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-16 23:05:14.534665300 +0000 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-16 23:02:42.099868877 +0000 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-16 23:02:42.099868877 +0000 2024-09-16T23:05:20.768 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-16 23:02:42.099868877 +0000 2024-09-16T23:05:20.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T23:05:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-16T23:05:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-16T23:05:20.831 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000216317 s, 2.4 MB/s 2024-09-16T23:05:20.832 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T23:05:20.888 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:05:20.889 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T23:05:20.889 INFO:tasks.ceph:Generating config... 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:20.890 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:20.891 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:20.891 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:20.891 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:20.891 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:20.891 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:20.891 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T23:05:20.891 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T23:05:20.891 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T23:05:20.891 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T23:05:20.891 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T23:05:20.891 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T23:05:20.891 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T23:05:20.892 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T23:05:20.892 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T23:05:20.893 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-16T23:05:20.893 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-16T23:05:20.979 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.keyring 2024-09-16T23:05:20.982 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-16T23:05:21.024 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T23:05:21.089 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.96')] 2024-09-16T23:05:21.089 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.96', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T23:05:21.090 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:21.090 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T23:05:21.145 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.96 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: generated fsid 788e50d2-621a-4a2a-b433-20948746ee44 2024-09-16T23:05:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:setting min_mon_release = quincy 2024-09-16T23:05:21.225 INFO:teuthology.orchestra.run.smithi096.stdout:epoch 0 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:fsid 788e50d2-621a-4a2a-b433-20948746ee44 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:last_changed 2024-09-16T23:05:21.228984+0000 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:created 2024-09-16T23:05:21.228984+0000 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:min_mon_release 17 (quincy) 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:election_strategy: 1 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:0: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.a 2024-09-16T23:05:21.226 INFO:teuthology.orchestra.run.smithi096.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-16T23:05:21.227 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T23:05:21.282 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 788e50d2-621a-4a2a-b433-20948746ee44... 2024-09-16T23:05:21.283 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T23:05:21.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-16T23:05:21.365 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-16T23:05:21.366 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T23:05:21.367 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.96 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = 788e50d2-621a-4a2a-b433-20948746ee44 2024-09-16T23:05:21.368 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-16T23:05:21.369 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.370 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = lz4 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-16T23:05:21.371 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-16T23:05:21.372 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T23:05:21.373 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-16T23:05:21.374 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: rbd mirror image policy type = none 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.375 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T23:05:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T23:05:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = 172.21.15.96 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-09-16T23:05:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 788e50d2-621a-4a2a-b433-20948746ee44 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-09-16T23:05:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression algorithm = lz4 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-09-16T23:05:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T23:05:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T23:05:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: debug rbd = 20 2024-09-16T23:05:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: debug rbd_mirror = 15 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: log to stderr = False 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: rbd default features = 125 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: rbd mirror image policy type = none 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.0] 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.1] 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.2] 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.3] 2024-09-16T23:05:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:21.388 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-16T23:05:21.388 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-16T23:05:21.482 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T23:05:21.482 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:21.482 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-16T23:05:21.497 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:21.497 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-16T23:05:21.552 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:05:21.552 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:21.552 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T23:05:21.553 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T23:05:21.626 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:21.626 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:21.681 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:05:21.682 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:21.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T23:05:21.682 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T23:05:21.722 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:21.722 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:21.779 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T23:05:21.780 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T23:05:21.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T23:05:21.830 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T23:05:21.833 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T23:05:21.833 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T23:05:21.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-16T23:05:21.876 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-16T23:05:21.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T23:05:21.945 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T23:05:21.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T23:05:22.002 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T23:05:22.015 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T23:05:22.097 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T23:05:22.109 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T23:05:22.190 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T23:05:22.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T23:05:22.284 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T23:05:22.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T23:05:22.378 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T23:05:22.391 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T23:05:22.472 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T23:05:22.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T23:05:22.566 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T23:05:22.578 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T23:05:22.578 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T23:05:22.579 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-16T23:05:22.606 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:05:22.606 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-16T23:05:22.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:05:22.607 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:05:22.680 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:05:22.681 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:05:23.184 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-16T23:05:23.186 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-16T23:05:23.186 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-16T23:05:23.334 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-16T23:05:23.367 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-16T23:05:23.432 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:05:23.432 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-16T23:05:23.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:05:23.432 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T23:05:23.506 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:05:23.506 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:05:23.506 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:05:23.506 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:05:23.507 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:05:23.959 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-16T23:05:23.961 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-16T23:05:23.962 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-16T23:05:24.056 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-16T23:05:24.085 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-16T23:05:24.152 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:05:24.152 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-16T23:05:24.152 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:05:24.152 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T23:05:24.225 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:05:24.225 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:05:24.226 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:05:24.710 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-16T23:05:24.712 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-16T23:05:24.712 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-16T23:05:24.849 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-16T23:05:24.879 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:24.972 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:24.975+0000 7f9ddb018640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-16T23:05:24.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:24.976+0000 7f9ddb018640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-16T23:05:24.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:24.977+0000 7f9ddb018640 -1 bdev(0x55b987b79800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-16T23:05:24.974 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:24.977+0000 7f9ddb018640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-16T23:05:27.897 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-16T23:05:27.927 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:28.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:28.021+0000 7f040d20d640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-16T23:05:28.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:28.021+0000 7f040d20d640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-16T23:05:28.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:28.021+0000 7f040d20d640 -1 bdev(0x55dd9927b800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-16T23:05:28.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:28.021+0000 7f040d20d640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-16T23:05:31.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-16T23:05:31.208 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:31.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:31.301+0000 7fec97a15640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-16T23:05:31.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:31.301+0000 7fec97a15640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-16T23:05:31.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:31.301+0000 7fec97a15640 -1 bdev(0x559d0cd8f800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-16T23:05:31.299 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-16T23:05:31.301+0000 7fec97a15640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-16T23:05:34.456 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-16T23:05:34.485 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T23:05:34.486 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:34.486 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-16T23:05:34.554 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:34.554 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-16T23:05:34.620 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:34.620 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-16T23:05:34.687 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:34.687 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-16T23:05:34.754 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:34.755 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-16T23:05:34.810 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:34.810 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-16T23:05:34.830 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:34.830 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-16T23:05:34.887 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:34.887 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-16T23:05:34.946 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:34.946 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-16T23:05:35.004 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:35.004 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-16T23:05:35.061 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:35.062 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-16T23:05:35.119 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:35.119 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-16T23:05:35.175 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:35.175 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-16T23:05:35.233 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T23:05:35.233 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCxuehmLtayMRAAUWznPB1UnOHtmX7BdSxlnA== 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC0uehmV9kwOhAAR7LdDmddAm+Gl2xBFMpHVA== 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC4uehmnMBRARAARfoaDX4tzlJgpOc7U6zWpA== 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC7uehm/cMDEhAAXn3qvsZ2KJklunB5gGCQ8w== 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-09-16T23:05:35.257 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCxuehmnMxONBAADk0suq/8gmEEkcNlb8F3jA== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCxuehmRy6WOBAARfWfei/qeJdwhwjirVQivw== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmoR5ZABAASGnf0eq2wh2kmbr5x/dcXw== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehm+pb+BRAAgVcTAtOiUmxnMBkcfPhcNw== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmpR+TCxAAFF/9S1CZgn/7QW7qMuVLww== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmybsiERAA8QaK4iBKm5QUJY/n9er4Hw== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.4] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmmB7BFhAAWfdR7b7+cKXHmg0t5EFJHA== 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.5] 2024-09-16T23:05:35.258 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmPFJlHBAAKiP8mgfL80oDLu2Z4tw6kA== 2024-09-16T23:05:35.259 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.6] 2024-09-16T23:05:35.259 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyuehmNl/xIRAA77bfqUIQmUhEZDZ9yduzaA== 2024-09-16T23:05:35.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T23:05:35.351 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:05:35.401 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:05:35.452 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:05:35.502 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.556 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.610 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.663 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.717 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.770 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.824 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.878 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.932 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:05:35.985 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T23:05:35.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-16T23:05:36.010 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-16T23:05:36.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-16T23:05:36.132 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T23:05:36.188 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-16T23:05:36.188 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-16T23:05:36.188 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-16T23:05:36.231 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-16T23:05:36.231 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-16T23:05:36.231 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-16T23:05:36.231 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-16T23:05:36.233 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-16T23:05:36.233 DEBUG:tasks.ceph:set 0 configs 2024-09-16T23:05:36.233 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T23:05:36.234 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-16T23:05:36.405 INFO:teuthology.orchestra.run.smithi096.stderr:adjusted tunables profile to default 2024-09-16T23:05:36.415 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T23:05:36.416 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-16T23:05:36.416 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:36.416 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-16T23:05:36.440 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:36.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-16T23:05:36.506 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:05:36.506 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-16T23:05:36.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new 12a56727-19bf-4ba9-ba93-67a8c3f83bd2 0 2024-09-16T23:05:36.788 INFO:teuthology.orchestra.run.smithi096.stdout:0 2024-09-16T23:05:36.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new cffe71c7-f3a6-474d-9b81-85f56cc55750 1 2024-09-16T23:05:36.975 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-09-16T23:05:36.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd new 5d99f04e-e5b2-4b72-b9d4-3d29868fbb40 2 2024-09-16T23:05:37.164 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-09-16T23:05:37.175 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-16T23:05:37.175 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-16T23:05:37.177 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-16T23:05:37.178 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-16T23:05:37.178 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-16T23:05:37.179 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-16T23:05:37.180 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-16T23:05:37.180 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-16T23:05:37.182 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-16T23:05:37.182 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:37.375 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:37.375 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:37.166708+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:37.385 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-16T23:05:37.385 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T23:05:37.688 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:37.847 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-16T23:05:37.847 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":5,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:37.166708+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:37.857 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T23:05:38.417 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:05:38.419+0000 7f4bd92d4640 -1 Falling back to public interface 2024-09-16T23:05:38.655 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:05:38.657+0000 7f8c1b73e640 -1 Falling back to public interface 2024-09-16T23:05:38.660 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-16T23:05:38.662+0000 7f021c287640 -1 Falling back to public interface 2024-09-16T23:05:41.694 INFO:tasks.ceph.cluster1.mgr.x.smithi096.stderr:2024-09-16T23:05:41.696+0000 7fe093f3a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T23:05:42.505 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:05:42.507+0000 7f8c1b73e640 -1 osd.0 0 log_to_monitors true 2024-09-16T23:05:42.572 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:05:42.574+0000 7f4bd92d4640 -1 osd.2 0 log_to_monitors true 2024-09-16T23:05:42.772 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-16T23:05:42.774+0000 7f021c287640 -1 osd.1 0 log_to_monitors true 2024-09-16T23:05:44.160 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:44.578 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-16T23:05:44.578 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":7,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:43.697606+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:44.590 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T23:05:44.610 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:05:44.612+0000 7f4bd5263640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T23:05:44.621 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:05:44.623+0000 7f4bcd082640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:05:44.645 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:05:44.647+0000 7f8c176cd640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T23:05:44.659 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:05:44.661+0000 7f8c0f4ec640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:05:45.418 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-16T23:05:45.420+0000 7f0218a2a640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T23:05:45.428 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-16T23:05:45.430+0000 7f0210035640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:05:50.893 DEBUG:teuthology.orchestra.run.smithi096:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:51.182 INFO:teuthology.misc.health.smithi096.stdout: 2024-09-16T23:05:51.182 INFO:teuthology.misc.health.smithi096.stdout:{"epoch":11,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:47.742707+0000","last_up_change":"2024-09-16T23:05:45.707878+0000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:05:44.711648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6809","nonce":1838959797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6811","nonce":1838959797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6815","nonce":1838959797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6813","nonce":1838959797}]},"public_addr":"172.21.15.96:6809/1838959797","cluster_addr":"172.21.15.96:6811/1838959797","heartbeat_back_addr":"172.21.15.96:6815/1838959797","heartbeat_front_addr":"172.21.15.96:6813/1838959797","state":["exists","up"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6817","nonce":3258003119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6819","nonce":3258003119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6823","nonce":3258003119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6821","nonce":3258003119}]},"public_addr":"172.21.15.96:6817/3258003119","cluster_addr":"172.21.15.96:6819/3258003119","heartbeat_back_addr":"172.21.15.96:6823/3258003119","heartbeat_front_addr":"172.21.15.96:6821/3258003119","state":["exists","up"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6801","nonce":1539697306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6803","nonce":1539697306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6807","nonce":1539697306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6805","nonce":1539697306}]},"public_addr":"172.21.15.96:6801/1539697306","cluster_addr":"172.21.15.96:6803/1539697306","heartbeat_back_addr":"172.21.15.96:6807/1539697306","heartbeat_front_addr":"172.21.15.96:6805/1539697306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.473113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.729249+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.626380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:51.194 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T23:05:51.194 INFO:tasks.ceph:Creating RBD pool 2024-09-16T23:05:51.194 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-16T23:05:51.751 INFO:teuthology.orchestra.run.smithi096.stderr:pool 'rbd' created 2024-09-16T23:05:51.767 DEBUG:teuthology.orchestra.run.smithi096:> rbd --cluster cluster1 pool init rbd 2024-09-16T23:05:54.773 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-16T23:05:54.774 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2024-09-16T23:05:54.774 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T23:05:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:55.078 INFO:teuthology.orchestra.run.smithi096.stdout:[{"version":4,"timestamp":"2024-09-16T23:05:45.422404+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"7135.063241"}]}] 2024-09-16T23:05:55.078 INFO:tasks.ceph_manager:config epoch is 4 2024-09-16T23:05:55.078 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-16T23:05:55.078 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-16T23:05:55.078 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-16T23:05:55.407 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:55.417 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":2842001185},{"type":"v1","addr":"172.21.15.96:6825","nonce":2842001185}]},"active_addr":"172.21.15.96:6825/2842001185","active_change":"2024-09-16T23:05:40.691387+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":749807968}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":937854459}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":2185820935}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":4253410158}]}]} 2024-09-16T23:05:55.420 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-16T23:05:55.420 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-16T23:05:55.420 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:55.700 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:55.701 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:54.761540+0000","last_up_change":"2024-09-16T23:05:45.707878+0000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:05:44.711648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:05:51.483810+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6809","nonce":1838959797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6811","nonce":1838959797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6815","nonce":1838959797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6813","nonce":1838959797}]},"public_addr":"172.21.15.96:6809/1838959797","cluster_addr":"172.21.15.96:6811/1838959797","heartbeat_back_addr":"172.21.15.96:6815/1838959797","heartbeat_front_addr":"172.21.15.96:6813/1838959797","state":["exists","up"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6817","nonce":3258003119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6819","nonce":3258003119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6823","nonce":3258003119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6821","nonce":3258003119}]},"public_addr":"172.21.15.96:6817/3258003119","cluster_addr":"172.21.15.96:6819/3258003119","heartbeat_back_addr":"172.21.15.96:6823/3258003119","heartbeat_front_addr":"172.21.15.96:6821/3258003119","state":["exists","up"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6801","nonce":1539697306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6803","nonce":1539697306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6807","nonce":1539697306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6805","nonce":1539697306}]},"public_addr":"172.21.15.96:6801/1539697306","cluster_addr":"172.21.15.96:6803/1539697306","heartbeat_back_addr":"172.21.15.96:6807/1539697306","heartbeat_front_addr":"172.21.15.96:6805/1539697306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.473113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.729249+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.626380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:55.713 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-16T23:05:55.713 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:05:55.998 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:55.998 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":15,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:05:54.761540+0000","last_up_change":"2024-09-16T23:05:45.707878+0000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:05:44.711648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:05:51.483810+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6809","nonce":1838959797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6811","nonce":1838959797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6815","nonce":1838959797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6813","nonce":1838959797}]},"public_addr":"172.21.15.96:6809/1838959797","cluster_addr":"172.21.15.96:6811/1838959797","heartbeat_back_addr":"172.21.15.96:6815/1838959797","heartbeat_front_addr":"172.21.15.96:6813/1838959797","state":["exists","up"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6817","nonce":3258003119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6819","nonce":3258003119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6823","nonce":3258003119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6821","nonce":3258003119}]},"public_addr":"172.21.15.96:6817/3258003119","cluster_addr":"172.21.15.96:6819/3258003119","heartbeat_back_addr":"172.21.15.96:6823/3258003119","heartbeat_front_addr":"172.21.15.96:6821/3258003119","state":["exists","up"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6801","nonce":1539697306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6803","nonce":1539697306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6807","nonce":1539697306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6805","nonce":1539697306}]},"public_addr":"172.21.15.96:6801/1539697306","cluster_addr":"172.21.15.96:6803/1539697306","heartbeat_back_addr":"172.21.15.96:6807/1539697306","heartbeat_front_addr":"172.21.15.96:6805/1539697306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.473113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.729249+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.626380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:05:56.010 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-16T23:05:56.011 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-16T23:05:56.011 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-16T23:05:56.150 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-16T23:05:56.150 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T23:05:56.153 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-16T23:05:56.154 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T23:05:56.154 INFO:teuthology.orchestra.run.smithi096.stdout:38654705669 2024-09-16T23:05:56.155 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T23:05:56.453 INFO:teuthology.orchestra.run.smithi096.stdout:34359738370 2024-09-16T23:05:56.465 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738370 for osd.0 2024-09-16T23:05:56.493 INFO:teuthology.orchestra.run.smithi096.stdout:34359738370 2024-09-16T23:05:56.500 INFO:teuthology.orchestra.run.smithi096.stdout:38654705668 2024-09-16T23:05:56.506 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738370 for osd.2 2024-09-16T23:05:56.513 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705668 for osd.1 2024-09-16T23:05:57.467 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T23:05:57.507 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T23:05:57.514 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T23:05:57.769 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-16T23:05:57.782 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.0 2024-09-16T23:05:57.782 DEBUG:teuthology.parallel:result is None 2024-09-16T23:05:57.812 INFO:teuthology.orchestra.run.smithi096.stdout:34359738372 2024-09-16T23:05:57.825 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2024-09-16T23:05:57.825 DEBUG:teuthology.parallel:result is None 2024-09-16T23:05:57.852 INFO:teuthology.orchestra.run.smithi096.stdout:38654705669 2024-09-16T23:05:57.865 INFO:tasks.ceph.ceph_manager.cluster1:need seq 38654705669 got 38654705669 for osd.1 2024-09-16T23:05:57.865 DEBUG:teuthology.parallel:result is None 2024-09-16T23:05:57.865 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T23:05:57.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:05:58.183 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:58.184 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:05:58.195 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T23:05:56.697898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.942603"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772641+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:05:54.772641+0000","last_peered":"2024-09-16T23:05:54.772641+0000","last_clean":"2024-09-16T23:05:54.772641+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:05:54.772641+0000","last_undegraded":"2024-09-16T23:05:54.772641+0000","last_fullsized":"2024-09-16T23:05:54.772641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772355+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:05:54.772355+0000","last_peered":"2024-09-16T23:05:54.772355+0000","last_clean":"2024-09-16T23:05:54.772355+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:05:54.772355+0000","last_undegraded":"2024-09-16T23:05:54.772355+0000","last_fullsized":"2024-09-16T23:05:54.772355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772387+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:05:54.772387+0000","last_peered":"2024-09-16T23:05:54.772387+0000","last_clean":"2024-09-16T23:05:54.772387+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:05:54.772387+0000","last_undegraded":"2024-09-16T23:05:54.772387+0000","last_fullsized":"2024-09-16T23:05:54.772387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772071+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:05:54.772071+0000","last_peered":"2024-09-16T23:05:54.772071+0000","last_clean":"2024-09-16T23:05:54.772071+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:05:54.772071+0000","last_undegraded":"2024-09-16T23:05:54.772071+0000","last_fullsized":"2024-09-16T23:05:54.772071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.770744+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:05:54.770744+0000","last_peered":"2024-09-16T23:05:54.770744+0000","last_clean":"2024-09-16T23:05:54.770744+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:05:54.770744+0000","last_undegraded":"2024-09-16T23:05:54.770744+0000","last_fullsized":"2024-09-16T23:05:54.770744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.765826+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:05:54.765826+0000","last_peered":"2024-09-16T23:05:54.765826+0000","last_clean":"2024-09-16T23:05:54.765826+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:05:54.765826+0000","last_undegraded":"2024-09-16T23:05:54.765826+0000","last_fullsized":"2024-09-16T23:05:54.765826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.765915+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:05:54.765915+0000","last_peered":"2024-09-16T23:05:54.765915+0000","last_clean":"2024-09-16T23:05:54.765915+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:05:54.765915+0000","last_undegraded":"2024-09-16T23:05:54.765915+0000","last_fullsized":"2024-09-16T23:05:54.765915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772010+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:05:54.772010+0000","last_peered":"2024-09-16T23:05:54.772010+0000","last_clean":"2024-09-16T23:05:54.772010+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:05:54.772010+0000","last_undegraded":"2024-09-16T23:05:54.772010+0000","last_fullsized":"2024-09-16T23:05:54.772010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.771401+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:05:54.771401+0000","last_peered":"2024-09-16T23:05:54.771401+0000","last_clean":"2024-09-16T23:05:54.771401+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:05:54.771401+0000","last_undegraded":"2024-09-16T23:05:54.771401+0000","last_fullsized":"2024-09-16T23:05:54.771401+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:05:58.196 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:05:58.475 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:58.475 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:05:58.487 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T23:05:56.697898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81592,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033928,"statfs":{"total":289910292480,"available":289826742272,"internally_reserved":0,"allocated":516096,"data_stored":1296850,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.942603"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772641+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:05:54.772641+0000","last_peered":"2024-09-16T23:05:54.772641+0000","last_clean":"2024-09-16T23:05:54.772641+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:05:54.772641+0000","last_undegraded":"2024-09-16T23:05:54.772641+0000","last_fullsized":"2024-09-16T23:05:54.772641+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772355+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:05:54.772355+0000","last_peered":"2024-09-16T23:05:54.772355+0000","last_clean":"2024-09-16T23:05:54.772355+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:05:54.772355+0000","last_undegraded":"2024-09-16T23:05:54.772355+0000","last_fullsized":"2024-09-16T23:05:54.772355+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772387+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:05:54.772387+0000","last_peered":"2024-09-16T23:05:54.772387+0000","last_clean":"2024-09-16T23:05:54.772387+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:05:54.772387+0000","last_undegraded":"2024-09-16T23:05:54.772387+0000","last_fullsized":"2024-09-16T23:05:54.772387+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772071+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:05:54.772071+0000","last_peered":"2024-09-16T23:05:54.772071+0000","last_clean":"2024-09-16T23:05:54.772071+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:05:54.772071+0000","last_undegraded":"2024-09-16T23:05:54.772071+0000","last_fullsized":"2024-09-16T23:05:54.772071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.770744+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:05:54.770744+0000","last_peered":"2024-09-16T23:05:54.770744+0000","last_clean":"2024-09-16T23:05:54.770744+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:05:54.770744+0000","last_undegraded":"2024-09-16T23:05:54.770744+0000","last_fullsized":"2024-09-16T23:05:54.770744+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.765826+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:05:54.765826+0000","last_peered":"2024-09-16T23:05:54.765826+0000","last_clean":"2024-09-16T23:05:54.765826+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:05:54.765826+0000","last_undegraded":"2024-09-16T23:05:54.765826+0000","last_fullsized":"2024-09-16T23:05:54.765826+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.765915+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:05:54.765915+0000","last_peered":"2024-09-16T23:05:54.765915+0000","last_clean":"2024-09-16T23:05:54.765915+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:05:54.765915+0000","last_undegraded":"2024-09-16T23:05:54.765915+0000","last_fullsized":"2024-09-16T23:05:54.765915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.772010+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:05:54.772010+0000","last_peered":"2024-09-16T23:05:54.772010+0000","last_clean":"2024-09-16T23:05:54.772010+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:05:54.772010+0000","last_undegraded":"2024-09-16T23:05:54.772010+0000","last_fullsized":"2024-09-16T23:05:54.772010+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:05:54.771401+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:05:54.771401+0000","last_peered":"2024-09-16T23:05:54.771401+0000","last_clean":"2024-09-16T23:05:54.771401+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:05:54.771401+0000","last_undegraded":"2024-09-16T23:05:54.771401+0000","last_fullsized":"2024-09-16T23:05:54.771401+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705669,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":131072,"data_stored":38692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":629079,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:05:58.487 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T23:05:58.487 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-16T23:05:58.488 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-16T23:05:58.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-16T23:05:58.805 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:05:58.805 INFO:teuthology.orchestra.run.smithi096.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T23:05:58.817 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-16T23:05:58.818 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T23:05:58.827 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-16T23:05:58.827 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T23:05:58.828 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656', 'branch': 'wip-dis-testing', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7907656', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 70, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 2961, 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'sleep_before_teardown': 0, 'subset': '66/100', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_6e8474a02f2b86353394076ab80c6f50ea28e361/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361', 'targets': {'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJburlteIPoXlMMAu7agtyMe9MjtZIG6fiph3BK2/YJ+1V0KJNOfhlvM6TVwnXFsZpYH57jLXZ37V4WGK/WVzw0=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBErEuV/FPcG3SWS+1PR+Jy91XcYhrZ8Hz+THycDjFSZFEK1ONydI009YoWnGW62UjkZtF2YEhMN/AlQBnyoykfw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '6e8474a02f2b86353394076ab80c6f50ea28e361'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_19:45:14', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T23:05:58.828 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T23:05:58.831 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T23:05:58.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T23:05:58.888 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T23:05:58.920 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:58.920 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T23:05:58.978 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T23:05:58.978 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-09-16T23:05:59.035 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T23:05:59.036 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:59.036 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 23:05:14.545390799 +0000 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 23:02:41.113320316 +0000 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 23:02:41.113320316 +0000 2024-09-16T23:05:59.037 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-16 23:02:41.113320316 +0000 2024-09-16T23:05:59.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T23:05:59.102 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T23:05:59.102 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T23:05:59.102 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000286885 s, 1.8 MB/s 2024-09-16T23:05:59.104 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T23:05:59.161 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 23:05:14.545390799 +0000 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 23:02:41.435316263 +0000 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 23:02:41.435316263 +0000 2024-09-16T23:05:59.219 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-16 23:02:41.435316263 +0000 2024-09-16T23:05:59.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T23:05:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T23:05:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T23:05:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000219099 s, 2.3 MB/s 2024-09-16T23:05:59.285 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T23:05:59.343 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 23:05:14.545390799 +0000 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 23:02:41.752312273 +0000 2024-09-16T23:05:59.398 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 23:02:41.752312273 +0000 2024-09-16T23:05:59.399 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-16 23:02:41.752312273 +0000 2024-09-16T23:05:59.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T23:05:59.463 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T23:05:59.463 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T23:05:59.463 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000196467 s, 2.6 MB/s 2024-09-16T23:05:59.464 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T23:05:59.520 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-09-16 23:05:14.546390787 +0000 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-09-16 23:02:42.074308219 +0000 2024-09-16T23:05:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-09-16 23:02:42.074308219 +0000 2024-09-16T23:05:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-09-16 23:02:42.074308219 +0000 2024-09-16T23:05:59.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T23:05:59.640 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-09-16T23:05:59.641 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-09-16T23:05:59.641 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000202507 s, 2.5 MB/s 2024-09-16T23:05:59.642 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T23:05:59.698 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:05:59.698 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T23:05:59.698 INFO:tasks.ceph:Generating config... 2024-09-16T23:05:59.699 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client] rbd mirror image policy type = none 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:05:59.700 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:05:59.700 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T23:05:59.701 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T23:05:59.701 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T23:05:59.701 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T23:05:59.701 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T23:05:59.701 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T23:05:59.701 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T23:05:59.701 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T23:05:59.701 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T23:05:59.702 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T23:05:59.702 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-16T23:05:59.702 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-16T23:05:59.778 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.keyring 2024-09-16T23:05:59.781 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-16T23:05:59.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T23:05:59.927 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.98')] 2024-09-16T23:05:59.928 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.98', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'none'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T23:05:59.928 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:05:59.929 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T23:05:59.983 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.98 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: generated fsid 9e3c1a33-dc76-489c-9850-c3603aaaec59 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 0 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 9e3c1a33-dc76-489c-9850-c3603aaaec59 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:last_changed 2024-09-16T23:06:00.067416+0000 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-09-16T23:06:00.067416+0000 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:min_mon_release 17 (quincy) 2024-09-16T23:06:00.064 INFO:teuthology.orchestra.run.smithi098.stdout:election_strategy: 1 2024-09-16T23:06:00.065 INFO:teuthology.orchestra.run.smithi098.stdout:0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-09-16T23:06:00.065 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-16T23:06:00.066 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T23:06:00.121 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 9e3c1a33-dc76-489c-9850-c3603aaaec59... 2024-09-16T23:06:00.121 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T23:06:00.124 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T23:06:00.172 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-16T23:06:00.172 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-16T23:06:00.172 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.173 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-16T23:06:00.174 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-16T23:06:00.175 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = 172.21.15.98 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = 9e3c1a33-dc76-489c-9850-c3603aaaec59 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-16T23:06:00.176 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.177 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = lz4 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:06:00.178 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:06:00.179 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.180 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-16T23:06:00.181 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd = 20 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: debug rbd_mirror = 15 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: log to stderr = False 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: rbd default features = 125 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: rbd mirror image policy type = none 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.0] 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.1] 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.182 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.2] 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout:[client.mirror.3] 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.183 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-09-16T23:06:00.203 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-09-16T23:06:00.204 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T23:06:00.205 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = 172.21.15.98 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 9e3c1a33-dc76-489c-9850-c3603aaaec59 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-09-16T23:06:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-09-16T23:06:00.207 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-09-16T23:06:00.208 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression algorithm = lz4 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore compression mode = aggressive 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1/20 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1/20 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 4/10 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-09-16T23:06:00.209 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-09-16T23:06:00.210 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T23:06:00.211 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: debug rbd = 20 2024-09-16T23:06:00.212 INFO:teuthology.orchestra.run.smithi098.stdout: debug rbd_mirror = 15 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: log to stderr = False 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: rbd default features = 125 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: rbd mirror image policy type = none 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.0] 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.1] 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.2] 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.3] 2024-09-16T23:06:00.213 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T23:06:00.214 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T23:06:00.214 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-16T23:06:00.214 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-16T23:06:00.309 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T23:06:00.309 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:00.310 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-16T23:06:00.324 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:00.325 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-16T23:06:00.379 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:06:00.380 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:06:00.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T23:06:00.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T23:06:00.419 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:06:00.419 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:00.476 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:06:00.476 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:00.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T23:06:00.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T23:06:00.511 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:00.511 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:00.566 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T23:06:00.566 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T23:06:00.566 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T23:06:00.622 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T23:06:00.624 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T23:06:00.625 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T23:06:00.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-16T23:06:00.667 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-16T23:06:00.680 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T23:06:00.720 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T23:06:00.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T23:06:00.813 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T23:06:00.826 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T23:06:00.908 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T23:06:00.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T23:06:01.000 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T23:06:01.013 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T23:06:01.093 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T23:06:01.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T23:06:01.186 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T23:06:01.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T23:06:01.280 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T23:06:01.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T23:06:01.372 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T23:06:01.385 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T23:06:01.385 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T23:06:01.385 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-16T23:06:01.450 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:06:01.450 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-16T23:06:01.451 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:06:01.451 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T23:06:01.522 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:06:01.523 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:06:02.030 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-16T23:06:02.032 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T23:06:02.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-16T23:06:02.171 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-16T23:06:02.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-16T23:06:02.271 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:06:02.272 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-16T23:06:02.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:06:02.272 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:06:02.345 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:06:02.797 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-16T23:06:02.799 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T23:06:02.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-16T23:06:02.893 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-16T23:06:02.924 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-16T23:06:02.990 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T23:06:02.990 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-16T23:06:02.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:06:02.991 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T23:06:03.064 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T23:06:03.064 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T23:06:03.064 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T23:06:03.065 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T23:06:03.570 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-09-16T23:06:03.572 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-09-16T23:06:03.573 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-16T23:06:03.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-16T23:06:03.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:03.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:03.857+0000 7fbee86fe640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-16T23:06:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:03.860+0000 7fbee86fe640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-16T23:06:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:03.860+0000 7fbee86fe640 -1 bdev(0x555ab26fb800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-16T23:06:03.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:03.860+0000 7fbee86fe640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-16T23:06:06.772 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-16T23:06:06.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:06.896+0000 7fada5422640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-16T23:06:06.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:06.896+0000 7fada5422640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-16T23:06:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:06.896+0000 7fada5422640 -1 bdev(0x55f9ac3e7800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-16T23:06:06.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:06.896+0000 7fada5422640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-16T23:06:10.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-16T23:06:10.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:10.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:10.161+0000 7f7c4260b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-16T23:06:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:10.161+0000 7f7c4260b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-16T23:06:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:10.162+0000 7f7c4260b640 -1 bdev(0x55fd74db5800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-16T23:06:10.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-09-16T23:06:10.162+0000 7f7c4260b640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-16T23:06:13.066 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-16T23:06:13.092 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T23:06:13.092 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:06:13.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-16T23:06:13.122 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.123 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-16T23:06:13.158 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.159 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-16T23:06:13.224 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-16T23:06:13.288 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-16T23:06:13.288 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-16T23:06:13.307 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.307 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-16T23:06:13.344 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.344 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-16T23:06:13.400 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.400 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-16T23:06:13.455 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.455 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-16T23:06:13.509 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.510 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-16T23:06:13.565 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.565 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-16T23:06:13.620 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.620 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-16T23:06:13.675 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:13.675 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-16T23:06:13.730 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T23:06:13.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm1ohKJRAAFpziIkAYf1SbRW96TNtcbg== 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDbuehmWPA1MxAAo3bj24Wk8g/VzzBxGqnDeA== 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-09-16T23:06:13.792 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDeuehm47CDNRAAHdUhAw58pegTZN635mOlOw== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDiuehm2C64CRAAjIxx0sjeezoKN79HtgxhCw== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehmfvD+JxAA/5LBlWjpbd7tQm4Xzbsi2w== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm0ConKxAAEQ4JuWZy+2lmx047BQLqYg== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.0] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm4tCzMBAAOiLLKdfXx0ZJwetq0Z7PSQ== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.1] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm+JRSNhAAbb+5cCvJJCsEFwVASTxs3w== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.2] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehmridCABAAmNNG9ce5wfaEDW5LANhDgA== 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.3] 2024-09-16T23:06:13.793 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehm7h3NBRAAJ1HKZOp6PqV6GoGaGM6WUg== 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.4] 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehml5xUCxAASaEtMTk6VciAuxbeR8oD0g== 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.5] 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehm8bnoEBAAKLdVGM1xXx/aiU/uKR5ckQ== 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.6] 2024-09-16T23:06:13.794 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehmZX1vFhAAQm33Abun1ebQhG3LbMBbFQ== 2024-09-16T23:06:13.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T23:06:13.886 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:06:13.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:06:13.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T23:06:14.038 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.144 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.465 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T23:06:14.518 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T23:06:14.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-16T23:06:14.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-16T23:06:14.637 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-16T23:06:14.661 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T23:06:14.717 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-16T23:06:14.717 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-16T23:06:14.717 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-16T23:06:14.759 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-16T23:06:14.759 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-16T23:06:14.759 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-16T23:06:14.760 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-16T23:06:14.762 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-16T23:06:14.762 DEBUG:tasks.ceph:set 0 configs 2024-09-16T23:06:14.762 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T23:06:14.762 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-16T23:06:14.930 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-09-16T23:06:14.942 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T23:06:14.942 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-16T23:06:14.942 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:14.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-16T23:06:14.967 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:14.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-16T23:06:15.031 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:06:15.031 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-16T23:06:15.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 osd new ba21fe61-6dc5-4802-90c4-e76f4cd45d47 0 2024-09-16T23:06:15.308 INFO:teuthology.orchestra.run.smithi098.stdout:0 2024-09-16T23:06:15.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 osd new 1427ee21-632b-49e9-9047-c174862d612f 1 2024-09-16T23:06:15.492 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-09-16T23:06:15.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 osd new 69605198-dc7b-4b62-b637-e200eff28561 2 2024-09-16T23:06:15.674 INFO:teuthology.orchestra.run.smithi098.stdout:2 2024-09-16T23:06:15.685 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-16T23:06:15.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-16T23:06:15.688 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-16T23:06:15.688 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-16T23:06:15.688 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-16T23:06:15.690 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-16T23:06:15.690 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-16T23:06:15.690 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-16T23:06:15.692 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-16T23:06:15.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:15.864 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:15.864 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:15.676979+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:15.873 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-16T23:06:15.873 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T23:06:16.175 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:16.337 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T23:06:16.337 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":5,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:15.676979+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:16.346 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T23:06:17.134 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:06:17.136+0000 7f441fe0d640 -1 Falling back to public interface 2024-09-16T23:06:17.370 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:06:17.372+0000 7fbd00219640 -1 Falling back to public interface 2024-09-16T23:06:17.377 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:06:17.379+0000 7f822841e640 -1 Falling back to public interface 2024-09-16T23:06:19.299 INFO:tasks.ceph.cluster2.mgr.x.smithi096.stderr:2024-09-16T23:06:19.301+0000 7fd69ba57640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T23:06:20.934 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:06:20.936+0000 7f441fe0d640 -1 osd.0 0 log_to_monitors true 2024-09-16T23:06:21.125 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:06:21.127+0000 7fbd00219640 -1 osd.2 0 log_to_monitors true 2024-09-16T23:06:21.173 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:06:21.175+0000 7f822841e640 -1 osd.1 0 log_to_monitors true 2024-09-16T23:06:22.650 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:23.068 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T23:06:23.068 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":7,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:22.304835+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:23.084 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T23:06:23.311 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:06:23.313+0000 7fbcfc415640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T23:06:23.311 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:06:23.313+0000 7f8224672640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T23:06:23.321 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:06:23.323+0000 7fbcf3a20640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:06:23.325 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:06:23.327+0000 7f441c05f640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T23:06:23.325 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:06:23.327+0000 7f821bc7d640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:06:23.336 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:06:23.338+0000 7f441366a640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T23:06:29.386 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:29.675 INFO:teuthology.misc.health.smithi098.stdout: 2024-09-16T23:06:29.675 INFO:teuthology.misc.health.smithi098.stdout:{"epoch":11,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:27.323824+0000","last_up_change":"2024-09-16T23:06:24.319317+0000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:06:24.460897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6801","nonce":1900552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6803","nonce":1900552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6807","nonce":1900552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6805","nonce":1900552820}]},"public_addr":"172.21.15.98:6801/1900552820","cluster_addr":"172.21.15.98:6803/1900552820","heartbeat_back_addr":"172.21.15.98:6807/1900552820","heartbeat_front_addr":"172.21.15.98:6805/1900552820","state":["exists","up"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6817","nonce":2210406205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6819","nonce":2210406205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6823","nonce":2210406205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6821","nonce":2210406205}]},"public_addr":"172.21.15.98:6817/2210406205","cluster_addr":"172.21.15.98:6819/2210406205","heartbeat_back_addr":"172.21.15.98:6823/2210406205","heartbeat_front_addr":"172.21.15.98:6821/2210406205","state":["exists","up"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6809","nonce":1071820272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6811","nonce":1071820272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6815","nonce":1071820272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6813","nonce":1071820272}]},"public_addr":"172.21.15.98:6809/1071820272","cluster_addr":"172.21.15.98:6811/1071820272","heartbeat_back_addr":"172.21.15.98:6815/1071820272","heartbeat_front_addr":"172.21.15.98:6813/1071820272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:21.951674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.170831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.115944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:29.686 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T23:06:29.686 INFO:tasks.ceph:Creating RBD pool 2024-09-16T23:06:29.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-16T23:06:30.330 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'rbd' created 2024-09-16T23:06:30.346 DEBUG:teuthology.orchestra.run.smithi098:> rbd --cluster cluster2 pool init rbd 2024-09-16T23:06:33.350 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-16T23:06:33.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2024-09-16T23:06:33.350 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T23:06:33.644 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:33.657 INFO:teuthology.orchestra.run.smithi098.stdout:[{"version":4,"timestamp":"2024-09-16T23:06:23.328801+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"4946.819212"}]}] 2024-09-16T23:06:33.657 INFO:tasks.ceph_manager:config epoch is 4 2024-09-16T23:06:33.657 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-16T23:06:33.657 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-16T23:06:33.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-16T23:06:33.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:33.997 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"flags":0,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":3852356284},{"type":"v1","addr":"172.21.15.96:6827","nonce":3852356284}]},"active_addr":"172.21.15.96:6827/3852356284","active_change":"2024-09-16T23:06:18.237941+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1772087855}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":227972451}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":337630091}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":4102645471}]}]} 2024-09-16T23:06:34.000 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-16T23:06:34.000 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-16T23:06:34.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:34.285 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:34.285 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:33.340737+0000","last_up_change":"2024-09-16T23:06:24.319317+0000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:06:24.460897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:06:29.974235+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6801","nonce":1900552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6803","nonce":1900552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6807","nonce":1900552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6805","nonce":1900552820}]},"public_addr":"172.21.15.98:6801/1900552820","cluster_addr":"172.21.15.98:6803/1900552820","heartbeat_back_addr":"172.21.15.98:6807/1900552820","heartbeat_front_addr":"172.21.15.98:6805/1900552820","state":["exists","up"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6817","nonce":2210406205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6819","nonce":2210406205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6823","nonce":2210406205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6821","nonce":2210406205}]},"public_addr":"172.21.15.98:6817/2210406205","cluster_addr":"172.21.15.98:6819/2210406205","heartbeat_back_addr":"172.21.15.98:6823/2210406205","heartbeat_front_addr":"172.21.15.98:6821/2210406205","state":["exists","up"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6809","nonce":1071820272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6811","nonce":1071820272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6815","nonce":1071820272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6813","nonce":1071820272}]},"public_addr":"172.21.15.98:6809/1071820272","cluster_addr":"172.21.15.98:6811/1071820272","heartbeat_back_addr":"172.21.15.98:6815/1071820272","heartbeat_front_addr":"172.21.15.98:6813/1071820272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:21.951674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.170831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.115944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:34.296 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-16T23:06:34.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:06:34.582 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:34.582 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":15,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:06:33.340737+0000","last_up_change":"2024-09-16T23:06:24.319317+0000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:06:24.460897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:06:29.974235+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6801","nonce":1900552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6803","nonce":1900552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6807","nonce":1900552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6805","nonce":1900552820}]},"public_addr":"172.21.15.98:6801/1900552820","cluster_addr":"172.21.15.98:6803/1900552820","heartbeat_back_addr":"172.21.15.98:6807/1900552820","heartbeat_front_addr":"172.21.15.98:6805/1900552820","state":["exists","up"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6817","nonce":2210406205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6819","nonce":2210406205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6823","nonce":2210406205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6821","nonce":2210406205}]},"public_addr":"172.21.15.98:6817/2210406205","cluster_addr":"172.21.15.98:6819/2210406205","heartbeat_back_addr":"172.21.15.98:6823/2210406205","heartbeat_front_addr":"172.21.15.98:6821/2210406205","state":["exists","up"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6809","nonce":1071820272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6811","nonce":1071820272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6815","nonce":1071820272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6813","nonce":1071820272}]},"public_addr":"172.21.15.98:6809/1071820272","cluster_addr":"172.21.15.98:6811/1071820272","heartbeat_back_addr":"172.21.15.98:6815/1071820272","heartbeat_front_addr":"172.21.15.98:6813/1071820272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:21.951674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.170831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.115944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:06:34.594 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-16T23:06:34.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-16T23:06:34.595 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-16T23:06:34.732 INFO:teuthology.orchestra.run.smithi098.stdout:34359738374 2024-09-16T23:06:34.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T23:06:34.752 INFO:teuthology.orchestra.run.smithi098.stdout:34359738373 2024-09-16T23:06:34.753 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T23:06:34.753 INFO:teuthology.orchestra.run.smithi098.stdout:34359738374 2024-09-16T23:06:34.754 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T23:06:35.038 INFO:teuthology.orchestra.run.smithi098.stdout:34359738373 2024-09-16T23:06:35.052 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-16T23:06:35.091 INFO:teuthology.orchestra.run.smithi098.stdout:34359738372 2024-09-16T23:06:35.100 INFO:teuthology.orchestra.run.smithi098.stdout:34359738373 2024-09-16T23:06:35.104 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.1 2024-09-16T23:06:35.113 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.2 2024-09-16T23:06:36.053 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T23:06:36.104 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T23:06:36.113 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T23:06:36.372 INFO:teuthology.orchestra.run.smithi098.stdout:34359738375 2024-09-16T23:06:36.389 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.0 2024-09-16T23:06:36.389 DEBUG:teuthology.parallel:result is None 2024-09-16T23:06:36.426 INFO:teuthology.orchestra.run.smithi098.stdout:34359738374 2024-09-16T23:06:36.440 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738374 for osd.1 2024-09-16T23:06:36.440 DEBUG:teuthology.parallel:result is None 2024-09-16T23:06:36.445 INFO:teuthology.orchestra.run.smithi098.stdout:34359738375 2024-09-16T23:06:36.458 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.2 2024-09-16T23:06:36.458 DEBUG:teuthology.parallel:result is None 2024-09-16T23:06:36.458 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T23:06:36.458 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:06:36.779 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:36.780 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:06:36.791 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T23:06:36.244962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81588,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033932,"statfs":{"total":289910292480,"available":289826746368,"internally_reserved":0,"allocated":516096,"data_stored":1295959,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.910478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349345+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:06:33.349345+0000","last_peered":"2024-09-16T23:06:33.349345+0000","last_clean":"2024-09-16T23:06:33.349345+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:06:33.349345+0000","last_undegraded":"2024-09-16T23:06:33.349345+0000","last_fullsized":"2024-09-16T23:06:33.349345+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349049+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:06:33.349049+0000","last_peered":"2024-09-16T23:06:33.349049+0000","last_clean":"2024-09-16T23:06:33.349049+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:06:33.349049+0000","last_undegraded":"2024-09-16T23:06:33.349049+0000","last_fullsized":"2024-09-16T23:06:33.349049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349406+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:06:33.349406+0000","last_peered":"2024-09-16T23:06:33.349406+0000","last_clean":"2024-09-16T23:06:33.349406+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:06:33.349406+0000","last_undegraded":"2024-09-16T23:06:33.349406+0000","last_fullsized":"2024-09-16T23:06:33.349406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349591+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:06:33.349591+0000","last_peered":"2024-09-16T23:06:33.349591+0000","last_clean":"2024-09-16T23:06:33.349591+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:06:33.349591+0000","last_undegraded":"2024-09-16T23:06:33.349591+0000","last_fullsized":"2024-09-16T23:06:33.349591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.347418+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:06:33.347418+0000","last_peered":"2024-09-16T23:06:33.347418+0000","last_clean":"2024-09-16T23:06:33.347418+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:06:33.347418+0000","last_undegraded":"2024-09-16T23:06:33.347418+0000","last_fullsized":"2024-09-16T23:06:33.347418+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.832856+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:06:33.832856+0000","last_peered":"2024-09-16T23:06:33.832856+0000","last_clean":"2024-09-16T23:06:33.832856+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:06:33.832856+0000","last_undegraded":"2024-09-16T23:06:33.832856+0000","last_fullsized":"2024-09-16T23:06:33.832856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.832838+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:06:33.832838+0000","last_peered":"2024-09-16T23:06:33.832838+0000","last_clean":"2024-09-16T23:06:33.832838+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:06:33.832838+0000","last_undegraded":"2024-09-16T23:06:33.832838+0000","last_fullsized":"2024-09-16T23:06:33.832838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349097+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:06:33.349097+0000","last_peered":"2024-09-16T23:06:33.349097+0000","last_clean":"2024-09-16T23:06:33.349097+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:06:33.349097+0000","last_undegraded":"2024-09-16T23:06:33.349097+0000","last_fullsized":"2024-09-16T23:06:33.349097+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.348229+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:06:33.348229+0000","last_peered":"2024-09-16T23:06:33.348229+0000","last_clean":"2024-09-16T23:06:33.348229+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:06:33.348229+0000","last_undegraded":"2024-09-16T23:06:33.348229+0000","last_fullsized":"2024-09-16T23:06:33.348229+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:06:36.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:06:37.072 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:37.073 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:06:37.084 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-16T23:06:36.244962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":81588,"kb_used_data":504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283033932,"statfs":{"total":289910292480,"available":289826746368,"internally_reserved":0,"allocated":516096,"data_stored":1295959,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.910478"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349345+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:06:33.349345+0000","last_peered":"2024-09-16T23:06:33.349345+0000","last_clean":"2024-09-16T23:06:33.349345+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:06:33.349345+0000","last_undegraded":"2024-09-16T23:06:33.349345+0000","last_fullsized":"2024-09-16T23:06:33.349345+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349049+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:06:33.349049+0000","last_peered":"2024-09-16T23:06:33.349049+0000","last_clean":"2024-09-16T23:06:33.349049+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:06:33.349049+0000","last_undegraded":"2024-09-16T23:06:33.349049+0000","last_fullsized":"2024-09-16T23:06:33.349049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349406+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:06:33.349406+0000","last_peered":"2024-09-16T23:06:33.349406+0000","last_clean":"2024-09-16T23:06:33.349406+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:06:33.349406+0000","last_undegraded":"2024-09-16T23:06:33.349406+0000","last_fullsized":"2024-09-16T23:06:33.349406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349591+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:06:33.349591+0000","last_peered":"2024-09-16T23:06:33.349591+0000","last_clean":"2024-09-16T23:06:33.349591+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:06:33.349591+0000","last_undegraded":"2024-09-16T23:06:33.349591+0000","last_fullsized":"2024-09-16T23:06:33.349591+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":22,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.347418+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:06:33.347418+0000","last_peered":"2024-09-16T23:06:33.347418+0000","last_clean":"2024-09-16T23:06:33.347418+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:06:33.347418+0000","last_undegraded":"2024-09-16T23:06:33.347418+0000","last_fullsized":"2024-09-16T23:06:33.347418+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.832856+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:06:33.832856+0000","last_peered":"2024-09-16T23:06:33.832856+0000","last_clean":"2024-09-16T23:06:33.832856+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:06:33.832856+0000","last_undegraded":"2024-09-16T23:06:33.832856+0000","last_fullsized":"2024-09-16T23:06:33.832856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.832838+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:06:33.832838+0000","last_peered":"2024-09-16T23:06:33.832838+0000","last_clean":"2024-09-16T23:06:33.832838+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:06:33.832838+0000","last_undegraded":"2024-09-16T23:06:33.832838+0000","last_fullsized":"2024-09-16T23:06:33.832838+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.349097+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:06:33.349097+0000","last_peered":"2024-09-16T23:06:33.349097+0000","last_clean":"2024-09-16T23:06:33.349097+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:06:33.349097+0000","last_undegraded":"2024-09-16T23:06:33.349097+0000","last_fullsized":"2024-09-16T23:06:33.349097+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'52","reported_seq":88,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T23:06:33.348229+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:06:33.348229+0000","last_peered":"2024-09-16T23:06:33.348229+0000","last_clean":"2024-09-16T23:06:33.348229+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:06:33.348229+0000","last_undegraded":"2024-09-16T23:06:33.348229+0000","last_fullsized":"2024-09-16T23:06:33.348229+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":628782,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:06:37.084 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T23:06:37.085 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-16T23:06:37.085 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-16T23:06:37.085 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-16T23:06:37.399 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:06:37.400 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T23:06:37.412 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-16T23:06:37.412 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T23:06:37.423 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T23:06:37.423 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:06:37.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:37.782 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror] 2024-09-16T23:06:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCxuehmRy6WOBAARfWfei/qeJdwhwjirVQivw== 2024-09-16T23:06:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-16T23:06:37.782 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:37.782 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror 2024-09-16T23:06:37.795 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:38.154 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.0] 2024-09-16T23:06:38.155 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCyuehmoR5ZABAASGnf0eq2wh2kmbr5x/dcXw== 2024-09-16T23:06:38.155 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:38.155 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:38.155 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.0 2024-09-16T23:06:38.168 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:38.529 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.1] 2024-09-16T23:06:38.530 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCyuehm+pb+BRAAgVcTAtOiUmxnMBkcfPhcNw== 2024-09-16T23:06:38.530 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:38.530 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:38.530 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.1 2024-09-16T23:06:38.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:38.908 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.2] 2024-09-16T23:06:38.908 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCyuehmpR+TCxAAFF/9S1CZgn/7QW7qMuVLww== 2024-09-16T23:06:38.908 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:38.908 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:38.908 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.2 2024-09-16T23:06:38.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:39.280 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.3] 2024-09-16T23:06:39.280 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQCyuehmybsiERAA8QaK4iBKm5QUJY/n9er4Hw== 2024-09-16T23:06:39.280 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:39.280 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.3 2024-09-16T23:06:39.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:39.659 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror] 2024-09-16T23:06:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm0ConKxAAEQ4JuWZy+2lmx047BQLqYg== 2024-09-16T23:06:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-16T23:06:39.659 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:39.659 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror 2024-09-16T23:06:39.673 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:40.038 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.0] 2024-09-16T23:06:40.038 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm4tCzMBAAOiLLKdfXx0ZJwetq0Z7PSQ== 2024-09-16T23:06:40.038 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:40.038 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.0 2024-09-16T23:06:40.052 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:40.416 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.1] 2024-09-16T23:06:40.416 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDYuehm+JRSNhAAbb+5cCvJJCsEFwVASTxs3w== 2024-09-16T23:06:40.416 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:40.417 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:40.417 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.1 2024-09-16T23:06:40.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:40.792 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.2] 2024-09-16T23:06:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehmridCABAAmNNG9ce5wfaEDW5LANhDgA== 2024-09-16T23:06:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:40.792 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:40.792 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.2 2024-09-16T23:06:40.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T23:06:41.169 INFO:teuthology.orchestra.run.smithi098.stdout:[client.mirror.3] 2024-09-16T23:06:41.170 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQDZuehm7h3NBRAAJ1HKZOp6PqV6GoGaGM6WUg== 2024-09-16T23:06:41.170 INFO:teuthology.orchestra.run.smithi098.stdout: caps mon = "profile rbd-mirror" 2024-09-16T23:06:41.170 INFO:teuthology.orchestra.run.smithi098.stdout: caps osd = "profile rbd" 2024-09-16T23:06:41.170 INFO:teuthology.orchestra.run.smithi098.stderr:updated caps for client.mirror.3 2024-09-16T23:06:41.183 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T23:06:41.195 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T23:06:41.198 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T23:06:41.198 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T23:06:41.210 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T23:06:41.213 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T23:06:41.213 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T23:06:41.220 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T23:06:41.222 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T23:06:41.222 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.230 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T23:06:41.230 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T23:06:41.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T23:06:41.232 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:06:41.240 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:06:41.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:06:41.242 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T23:06:41.250 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T23:06:41.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T23:06:41.253 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:06:41.261 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:06:41.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:06:41.263 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-16T23:06:41.272 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T23:06:41.272 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T23:06:41.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T23:06:41.275 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T23:06:41.285 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726528001 2024-09-16T23:06:41.285 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T23:06:41.285 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-16T23:06:41.295 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726528001 2024-09-16T23:06:41.295 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-16T23:06:41.295 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T23:06:41.307 INFO:tasks.workunit:Pulling workunits from ref 6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:06:41.307 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T23:06:41.307 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T23:06:41.308 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-16T23:06:41.308 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-16T23:06:41.308 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-16T23:06:41.308 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2024-09-16T23:06:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T23:06:41.330 INFO:teuthology.orchestra.run.smithi098.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-16T23:06:41.330 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T23:06:41.385 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T23:06:41.385 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-16T23:06:41.439 INFO:tasks.workunit:timeout=6h 2024-09-16T23:06:41.439 INFO:tasks.workunit:cleanup=True 2024-09-16T23:06:41.440 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 6e8474a02f2b86353394076ab80c6f50ea28e361 2024-09-16T23:06:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Note: switching to '6e8474a02f2b86353394076ab80c6f50ea28e361'. 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: git switch -c 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Or undo this operation with: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: git switch - 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:07:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:HEAD is now at 6e8474a02f2 Merge branch 'wip-nbalacha-ns-mirroring' of https://github.com/nbalacha/ceph into HEAD 2024-09-16T23:07:31.634 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-16T23:07:31.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T23:07:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T23:07:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T23:07:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T23:07:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T23:07:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T23:07:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T23:07:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T23:07:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T23:07:32.562 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-09-16T23:07:32.562 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-16T23:07:32.578 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-16T23:07:32.579 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-16T23:07:32.579 DEBUG:teuthology.orchestra.run.smithi098:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6e8474a02f2b86353394076ab80c6f50ea28e361 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T23:07:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ IMAGE_COUNT=50 2024-09-16T23:07:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ export LOCKDEP=0 2024-09-16T23:07:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ LOCKDEP=0 2024-09-16T23:07:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T23:07:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CLUSTER1=cluster1 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CLUSTER2=cluster2 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ POOL=mirror 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ PARENT_POOL=mirror_parent 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ NS1=ns1 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ NS2=ns2 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ TEMPDIR= 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ID=mirror 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ RBD_IMAGE_FEATURES=layering 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-16T23:07:32.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ MIRROR_POOL_MODE=image 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' snapshot = snapshot ']' 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ MIRROR_POOL_MODE=image 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS='--id mirror' 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-16T23:07:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-16T23:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-16T23:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_BIN=. 2024-09-16T23:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_SRC=. 2024-09-16T23:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-16T23:07:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 -gt 0 ']' 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ setup 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local c 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ setup_tempdir 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mktemp -d 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ TEMPDIR=/tmp/tmp.JxTgO6em7N 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z 1 ']' 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ setup_pools cluster1 cluster2 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster2 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mon_map_file 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mon_addr 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local admin_key_file 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid 2024-09-16T23:07:32.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-16T23:07:33.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror' already exists 2024-09-16T23:07:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-16T23:07:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror_parent' already exists 2024-09-16T23:07:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-16T23:07:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-16T23:07:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:40.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-16T23:07:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-16T23:07:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-16T23:07:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-16T23:07:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-16T23:07:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z ']' 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z ']' 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_add cluster1 mirror cluster2 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local client_cluster=cluster2 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster2 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid_var_name 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local error_code 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local peer_uuid 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:07:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_uuid=1b47ee54-18e0-4e20-bfd1-1a7d61f9026f 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 -eq 0 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-16T23:07:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror_parent 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local client_cluster=cluster2 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster2 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid_var_name 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local error_code 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local peer_uuid 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:07:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-16T23:07:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_uuid=c97cf795-5922-47de-b80b-c3cdd7e60201 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 -eq 0 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ setup_pools cluster2 cluster1 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster1 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mon_map_file 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mon_addr 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local admin_key_file 2024-09-16T23:07:40.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid 2024-09-16T23:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-16T23:07:41.325 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:07:41.327+0000 7f6af9e15880 -1 rbd::mirror::PoolReplayer: 0x5637a82e6c00 init: error accessing remote pool mirror: (2) No such file or directory 2024-09-16T23:07:41.344 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:07:41.346+0000 7f212c934880 -1 rbd::mirror::PoolReplayer: 0x559f9694ec00 init: error accessing remote pool mirror: (2) No such file or directory 2024-09-16T23:07:41.368 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:07:41.370+0000 7f09860a8880 -1 rbd::mirror::PoolReplayer: 0x559714b44c00 init: error accessing remote pool mirror: (2) No such file or directory 2024-09-16T23:07:41.375 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:07:41.377+0000 7feea2d75880 -1 rbd::mirror::PoolReplayer: 0x5595d1f22c00 init: error accessing remote pool mirror: (2) No such file or directory 2024-09-16T23:07:41.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:07:41.397+0000 7f212c934880 -1 rbd::mirror::PoolReplayer: 0x559f96a3af00 init: error accessing remote pool mirror_parent: (2) No such file or directory 2024-09-16T23:07:41.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:07:41.423+0000 7feea2d75880 -1 rbd::mirror::PoolReplayer: 0x5595d2c28f00 init: error accessing remote pool mirror_parent: (2) No such file or directory 2024-09-16T23:07:41.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror' already exists 2024-09-16T23:07:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-16T23:07:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror_parent' already exists 2024-09-16T23:07:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-16T23:07:43.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:07:43.438+0000 7f6af9e15880 -1 rbd::mirror::PoolReplayer: 0x5637a9044f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T23:07:43.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:07:43.441+0000 7f09860a8880 -1 rbd::mirror::PoolReplayer: 0x559714c38f00 init: failed to initialize remote pool poller: (2) No such file or directory 2024-09-16T23:07:45.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-16T23:07:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:07:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-16T23:07:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-16T23:07:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-16T23:07:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-16T23:07:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-16T23:07:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z ']' 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z ']' 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_add cluster2 mirror cluster1 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local client_cluster=cluster1 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster1 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid_var_name 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local error_code 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local peer_uuid 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:07:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_uuid=dcc71b61-58d3-4bc6-a6f7-eca95f6f74e9 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 -eq 0 2024-09-16T23:07:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror_parent 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local client_cluster=cluster1 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_cluster=cluster1 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local uuid_var_name 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local error_code 2024-09-16T23:07:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local peer_uuid 2024-09-16T23:07:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T23:07:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:07:48.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-16T23:07:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ peer_uuid=d7b7e755-e347-4d3a-903e-e4bcc4edb94e 2024-09-16T23:07:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:07:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:07:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 -eq 0 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirrors cluster1 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 0 3 2024-09-16T23:07:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster1:0 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:0 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster1:0 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster1 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=0 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster1 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:48.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=0 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster1:1 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:1 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster1:1 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster1 2024-09-16T23:07:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster1:2 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:2 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster1:2 2024-09-16T23:07:48.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster1 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=2 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster1 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=2 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster1:3 2024-09-16T23:07:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:3 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster1:3 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster1 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=3 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster1 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=3 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirrors cluster2 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 0 3 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster2:0 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:0 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster2:0 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster2 2024-09-16T23:07:48.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=0 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster2 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster2 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=0 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster2:1 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:1 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-16T23:07:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster2:1 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster2 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=1 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster2 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster2 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=1 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster2:2 2024-09-16T23:07:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster2:2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster2:3 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:3 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster2:3 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster2 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=3 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster2 2024-09-16T23:07:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster2 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=3 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:07:48 TEST: add image and test replay after client crashes 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:07:48 TEST: add image and test replay after client crashes 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=test 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror test 512M 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:07:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:07:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:07:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:07:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2024-09-16T23:07:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:07:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:07:48.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-16T23:07:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:07:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10bb7a87507d 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10bb7a87507d 2024-09-16T23:07:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:07:48 2024 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:07:48 2024 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:07:48 2024 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:07:49.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:07:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:07:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:07:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:07:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:07:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:07:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:07:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:07:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:07:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:07:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:07:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:50.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:07:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:07:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:07:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:07:50.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:07:50.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:07:50.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:07:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T23:07:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:50.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:3 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:3 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=3 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:3 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:3 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=3 2024-09-16T23:07:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=3 2024-09-16T23:07:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T23:07:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T23:07:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:50.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:50.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 1 2024-09-16T23:07:51.309 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:07:51.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:07:51.310 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T23:07:51.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:07:51.313+0000 7f3e25a17640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36955) UID: 1000 2024-09-16T23:07:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:07:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:07:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:07:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:07:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:07:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:52.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:52.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:07:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:07:52.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:07:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:07:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:07:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:07:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:07:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T23:07:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:52.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:3 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:3 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:52.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=3 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:3 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:3 2024-09-16T23:07:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=3 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=3 2024-09-16T23:07:52.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T23:07:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:52.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T23:07:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 2 2024-09-16T23:07:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:07:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:07:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:07:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:07:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:07:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:07:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:07:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:07:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:54.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:07:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:07:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:07:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:54.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:07:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:07:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:07:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:07:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T23:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:3 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:3 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=3 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:3 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:3 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=3 2024-09-16T23:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=3 2024-09-16T23:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T23:07:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:54.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T23:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 4 2024-09-16T23:07:58.311 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:07:58.312 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:07:58.312 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:07:58.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:07:58.312 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:07:58.315 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:07:58.315 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T23:07:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:07:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:07:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:07:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:07:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:07:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:07:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:07:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:07:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:07:58.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:07:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:58.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:07:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:07:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:07:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-16T23:07:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:07:58.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:3 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:3 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=3 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:3 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:3 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:07:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T23:07:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:07:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:07:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=3 2024-09-16T23:07:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=3 2024-09-16T23:07:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T23:07:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:07:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-16T23:07:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:07:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:07:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:07:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:07:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 8 2024-09-16T23:07:59.308 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.49589925610011376) > weight (0.25) 2024-09-16T23:07:59.308 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T23:07:59.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T23:07:59.309 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T23:07:59.309 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:07:59.313+0000 7f21298d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 36906) UID: 1000 2024-09-16T23:08:00.309 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T23:08:00.309 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:08:00.310 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T23:08:00.310 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T23:08:00.310 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T23:08:00.312 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T23:08:00.313 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-16T23:08:07.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:08:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:08:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:08:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:08:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:08:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:08:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:08:07.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:08:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:08:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:08:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:08:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:08:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:08:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:08:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:08:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:08:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:08:07.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:08:07.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:08:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:08:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:08:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:08:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:08:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:08:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:08:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:08:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:08:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:08:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name= 2024-09-16T23:08:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 10 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:08:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:08:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:08:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:08:07.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 68575232 bytes 2024-09-16T23:08:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 83799040 bytes 2024-09-16T23:08:07.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 127279104 bytes 2024-09-16T23:08:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 127896576 bytes 2024-09-16T23:08:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 136584192 bytes 2024-09-16T23:08:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 202959872 bytes 2024-09-16T23:08:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 229611520 bytes 2024-09-16T23:08:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 372712448 bytes 2024-09-16T23:08:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 414332928 bytes 2024-09-16T23:08:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 441527296 bytes 2024-09-16T23:08:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 477489152 bytes 2024-09-16T23:08:08.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 500097024 bytes 2024-09-16T23:08:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 557599744 bytes 2024-09-16T23:08:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 597779456 bytes 2024-09-16T23:08:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 677202944 bytes 2024-09-16T23:08:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 754210816 bytes 2024-09-16T23:08:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 797271040 bytes 2024-09-16T23:08:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 830703616 bytes 2024-09-16T23:08:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 831328256 bytes 2024-09-16T23:08:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 862041088 bytes 2024-09-16T23:08:09.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 888139776 bytes 2024-09-16T23:08:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 941859840 bytes 2024-09-16T23:08:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 957719552 bytes 2024-09-16T23:08:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 986852352 bytes 2024-09-16T23:08:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1051018240 bytes 2024-09-16T23:08:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1127046144 bytes 2024-09-16T23:08:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1149875200 bytes 2024-09-16T23:08:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1227143168 bytes 2024-09-16T23:08:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1257435136 bytes 2024-09-16T23:08:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1276816384 bytes 2024-09-16T23:08:09.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1305349120 bytes 2024-09-16T23:08:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1380518912 bytes 2024-09-16T23:08:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1436698624 bytes 2024-09-16T23:08:10.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1448609792 bytes 2024-09-16T23:08:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1501455360 bytes 2024-09-16T23:08:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1545826304 bytes 2024-09-16T23:08:10.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1605689344 bytes 2024-09-16T23:08:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1651276800 bytes 2024-09-16T23:08:10.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1709561856 bytes 2024-09-16T23:08:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1729270784 bytes 2024-09-16T23:08:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1771839488 bytes 2024-09-16T23:08:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1777857536 bytes 2024-09-16T23:08:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1846205440 bytes 2024-09-16T23:08:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1886376960 bytes 2024-09-16T23:08:11.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1949685760 bytes 2024-09-16T23:08:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1956839424 bytes 2024-09-16T23:08:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2004346880 bytes 2024-09-16T23:08:11.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:08:11.432+0000 7feea2d75880 -1 rbd::mirror::Mirror: 0x5595d11cfe60 update_pool_replayers: restarting failed pool replayer for uuid: 1b47ee54-18e0-4e20-bfd1-1a7d61f9026f cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.444 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:08:11.447+0000 7f6af9e15880 -1 rbd::mirror::Mirror: 0x5637a75a1e60 update_pool_replayers: restarting failed pool replayer for uuid: 1b47ee54-18e0-4e20-bfd1-1a7d61f9026f cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:08:11.450+0000 7f09860a8880 -1 rbd::mirror::Mirror: 0x559713cf5e60 update_pool_replayers: restarting failed pool replayer for uuid: 1b47ee54-18e0-4e20-bfd1-1a7d61f9026f cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2024478720 bytes 2024-09-16T23:08:11.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:08:11.534+0000 7feea2d75880 -1 rbd::mirror::Mirror: 0x5595d11cfe60 update_pool_replayers: restarting failed pool replayer for uuid: c97cf795-5922-47de-b80b-c3cdd7e60201 cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:08:11.544+0000 7f6af9e15880 -1 rbd::mirror::Mirror: 0x5637a75a1e60 update_pool_replayers: restarting failed pool replayer for uuid: c97cf795-5922-47de-b80b-c3cdd7e60201 cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.558 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:08:11.560+0000 7f09860a8880 -1 rbd::mirror::Mirror: 0x559713cf5e60 update_pool_replayers: restarting failed pool replayer for uuid: c97cf795-5922-47de-b80b-c3cdd7e60201 cluster: cluster2 client: client.mirror 2024-09-16T23:08:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2084698112 bytes 2024-09-16T23:08:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2107230208 bytes 2024-09-16T23:08:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2125426688 bytes 2024-09-16T23:08:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2186578944 bytes 2024-09-16T23:08:12.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2251187200 bytes 2024-09-16T23:08:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2310797312 bytes 2024-09-16T23:08:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2390629376 bytes 2024-09-16T23:08:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2409868288 bytes 2024-09-16T23:08:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2434173952 bytes 2024-09-16T23:08:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2468449280 bytes 2024-09-16T23:08:12.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2501067776 bytes 2024-09-16T23:08:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2501690368 bytes 2024-09-16T23:08:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2526382080 bytes 2024-09-16T23:08:12.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2585652224 bytes 2024-09-16T23:08:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2599307264 bytes 2024-09-16T23:08:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2662436864 bytes 2024-09-16T23:08:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2684861440 bytes 2024-09-16T23:08:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2778102784 bytes 2024-09-16T23:08:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2821513216 bytes 2024-09-16T23:08:13.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2825520128 bytes 2024-09-16T23:08:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2841174016 bytes 2024-09-16T23:08:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2868427776 bytes 2024-09-16T23:08:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2906896384 bytes 2024-09-16T23:08:13.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2981194752 bytes 2024-09-16T23:08:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3076481024 bytes 2024-09-16T23:08:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3116690432 bytes 2024-09-16T23:08:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3201240064 bytes 2024-09-16T23:08:14.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3214669824 bytes 2024-09-16T23:08:14.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3256841216 bytes 2024-09-16T23:08:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3318759424 bytes 2024-09-16T23:08:14.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3426861056 bytes 2024-09-16T23:08:14.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3497910272 bytes 2024-09-16T23:08:14.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3529588736 bytes 2024-09-16T23:08:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3597718528 bytes 2024-09-16T23:08:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3608918016 bytes 2024-09-16T23:08:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3618855936 bytes 2024-09-16T23:08:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3649555456 bytes 2024-09-16T23:08:15.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3650188288 bytes 2024-09-16T23:08:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3657732096 bytes 2024-09-16T23:08:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3660939264 bytes 2024-09-16T23:08:15.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3665201152 bytes 2024-09-16T23:08:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3752203264 bytes 2024-09-16T23:08:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3795972096 bytes 2024-09-16T23:08:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3845893120 bytes 2024-09-16T23:08:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3886576640 bytes 2024-09-16T23:08:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3930572800 bytes 2024-09-16T23:08:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3952254976 bytes 2024-09-16T23:08:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3955316736 bytes 2024-09-16T23:08:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4019012608 bytes 2024-09-16T23:08:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4076095488 bytes 2024-09-16T23:08:16.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4154899456 bytes 2024-09-16T23:08:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4174851072 bytes 2024-09-16T23:08:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4250021888 bytes 2024-09-16T23:08:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4282294272 bytes 2024-09-16T23:08:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4314402816 bytes 2024-09-16T23:08:16.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4374252544 bytes 2024-09-16T23:08:16.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4379825152 bytes 2024-09-16T23:08:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4407570432 bytes 2024-09-16T23:08:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4418061312 bytes 2024-09-16T23:08:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4474775552 bytes 2024-09-16T23:08:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4485501952 bytes 2024-09-16T23:08:17.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4499244032 bytes 2024-09-16T23:08:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4531006464 bytes 2024-09-16T23:08:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4563428352 bytes 2024-09-16T23:08:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4617923584 bytes 2024-09-16T23:08:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4638003200 bytes 2024-09-16T23:08:17.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 4654852096 bytes 2024-09-16T23:08:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 4725263360 bytes 2024-09-16T23:08:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 4743465984 bytes 2024-09-16T23:08:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 4779541504 bytes 2024-09-16T23:08:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 4788847616 bytes 2024-09-16T23:08:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 4832317440 bytes 2024-09-16T23:08:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 4848178176 bytes 2024-09-16T23:08:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 4924525568 bytes 2024-09-16T23:08:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12300 IOs, 4971838464 bytes 2024-09-16T23:08:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12400 IOs, 4998076416 bytes 2024-09-16T23:08:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12500 IOs, 4999747584 bytes 2024-09-16T23:08:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12600 IOs, 5000365056 bytes 2024-09-16T23:08:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12700 IOs, 5095299072 bytes 2024-09-16T23:08:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12800 IOs, 5170968576 bytes 2024-09-16T23:08:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12900 IOs, 5262529536 bytes 2024-09-16T23:08:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13000 IOs, 5306095616 bytes 2024-09-16T23:08:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13100 IOs, 5320056832 bytes 2024-09-16T23:08:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13200 IOs, 5389425664 bytes 2024-09-16T23:08:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13300 IOs, 5489395712 bytes 2024-09-16T23:08:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13400 IOs, 5500924928 bytes 2024-09-16T23:08:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13500 IOs, 5526634496 bytes 2024-09-16T23:08:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13600 IOs, 5599453184 bytes 2024-09-16T23:08:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13700 IOs, 5618908160 bytes 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:08:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:08:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:08:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:08:22.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:08:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: down+unknown 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: status not found 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: down+unknown 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: status not found 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: ' 2024-09-16T23:08:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:08:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 1 2024-09-16T23:08:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:08:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:08:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: down+unknown 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: status not found 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: down+unknown 2024-09-16T23:08:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: status not found 2024-09-16T23:08:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: ' 2024-09-16T23:08:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:08:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 1 2024-09-16T23:08:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:08:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:08:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:08:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:08:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: down+unknown 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: status not found 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:08:28' 2024-09-16T23:08:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 1 2024-09-16T23:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:28.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:08:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:08:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:08:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:08:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528069,"remote_snapshot_timestamp":1726528069,"replay_state":"idle"} 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:08:30 2024-09-16T23:08:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:08:28' 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:08:36.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:08:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:08:36.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528069,"remote_snapshot_timestamp":1726528069,"replay_state":"idle"} 2024-09-16T23:08:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:08:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:08:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:08:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:08:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:08:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:08:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:08:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:08:30' 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:08:30' 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:08:30' 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:08:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:08:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42204317mirror.2smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528069,"remote_snapshot_timestamp":1726528069,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:08:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:08:28' 2024-09-16T23:08:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:08:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:08:36.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:08:30' 2024-09-16T23:08:36.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:08:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:08:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:08:36.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:08:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap1 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap1 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1 2024-09-16T23:08:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:08:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:08:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:08:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:08:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:08:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:08:49.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:08:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:08:49.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:08:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:08:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:08:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:08:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:08:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:08:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:08:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:08:49.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:08:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:08:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:08:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:08:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:08:49.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:08:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:08:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:08:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:08:49.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:08:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:08:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:08:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:08:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:08:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:08:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:08:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:08:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:08:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:08:50.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 5 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:08:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:08:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:08:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=5 2024-09-16T23:08:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:08:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:08:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:08:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:08:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:08:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:08:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:08:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:08:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:08:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:08:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:08:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 5 = '' 2024-09-16T23:08:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:08:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:08:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:08:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:08:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:08:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:08:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:08:51.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:08:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:08:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 5 = '' 2024-09-16T23:08:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:08:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:08:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:51.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:08:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:08:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:08:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=5 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 5 = 5 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:08:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:08:52.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:08:53.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:08:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:08:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:53.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:08:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:08:53.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:08:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:08:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:08:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:08:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap1-clean 2024-09-16T23:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-16T23:08:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1-clean 2024-09-16T23:08:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:08:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:08:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:08:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:08:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:08:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:08:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:08:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:08:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:08:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:08:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 11904000 bytes 2024-09-16T23:08:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 31783936 bytes 2024-09-16T23:08:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 78950400 bytes 2024-09-16T23:08:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 89144320 bytes 2024-09-16T23:08:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 92310528 bytes 2024-09-16T23:08:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 124401664 bytes 2024-09-16T23:08:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 198228992 bytes 2024-09-16T23:08:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 243679232 bytes 2024-09-16T23:08:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 286638080 bytes 2024-09-16T23:08:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 322774016 bytes 2024-09-16T23:08:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 378556416 bytes 2024-09-16T23:08:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 379166720 bytes 2024-09-16T23:08:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 467805184 bytes 2024-09-16T23:08:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 521889792 bytes 2024-09-16T23:08:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 597642240 bytes 2024-09-16T23:08:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 628616192 bytes 2024-09-16T23:08:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 742126592 bytes 2024-09-16T23:08:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 772677632 bytes 2024-09-16T23:08:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 805192704 bytes 2024-09-16T23:08:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 829123584 bytes 2024-09-16T23:08:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 867647488 bytes 2024-09-16T23:08:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 918099968 bytes 2024-09-16T23:08:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1004555264 bytes 2024-09-16T23:08:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1028017152 bytes 2024-09-16T23:08:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1048671232 bytes 2024-09-16T23:08:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1123617792 bytes 2024-09-16T23:08:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1164848128 bytes 2024-09-16T23:08:58.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1211227136 bytes 2024-09-16T23:08:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1265340416 bytes 2024-09-16T23:08:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1303607296 bytes 2024-09-16T23:08:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1304228864 bytes 2024-09-16T23:08:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1328187392 bytes 2024-09-16T23:08:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1373771776 bytes 2024-09-16T23:08:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1448556544 bytes 2024-09-16T23:08:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1485070336 bytes 2024-09-16T23:08:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1510206464 bytes 2024-09-16T23:08:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1577438208 bytes 2024-09-16T23:08:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1624320000 bytes 2024-09-16T23:08:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1719432192 bytes 2024-09-16T23:08:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1754661888 bytes 2024-09-16T23:08:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1779277824 bytes 2024-09-16T23:08:59.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1817424896 bytes 2024-09-16T23:08:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1854581760 bytes 2024-09-16T23:08:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1953875968 bytes 2024-09-16T23:08:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2003352576 bytes 2024-09-16T23:08:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2004756480 bytes 2024-09-16T23:08:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2058336256 bytes 2024-09-16T23:08:59.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2119078912 bytes 2024-09-16T23:08:59.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2164673536 bytes 2024-09-16T23:09:00.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2192628736 bytes 2024-09-16T23:09:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2259277824 bytes 2024-09-16T23:09:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2298244096 bytes 2024-09-16T23:09:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2373180416 bytes 2024-09-16T23:09:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2400044032 bytes 2024-09-16T23:09:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2413806592 bytes 2024-09-16T23:09:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2482837504 bytes 2024-09-16T23:09:00.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2599152640 bytes 2024-09-16T23:09:00.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2635960320 bytes 2024-09-16T23:09:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2659390464 bytes 2024-09-16T23:09:01.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2757838848 bytes 2024-09-16T23:09:01.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2789684224 bytes 2024-09-16T23:09:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2852772864 bytes 2024-09-16T23:09:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2883723264 bytes 2024-09-16T23:09:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2906749952 bytes 2024-09-16T23:09:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2907353088 bytes 2024-09-16T23:09:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2907967488 bytes 2024-09-16T23:09:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2968878080 bytes 2024-09-16T23:09:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3048810496 bytes 2024-09-16T23:09:02.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3104439296 bytes 2024-09-16T23:09:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3151700992 bytes 2024-09-16T23:09:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3179171840 bytes 2024-09-16T23:09:02.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3203718144 bytes 2024-09-16T23:09:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3244005376 bytes 2024-09-16T23:09:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3275130880 bytes 2024-09-16T23:09:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3342383104 bytes 2024-09-16T23:09:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3391698944 bytes 2024-09-16T23:09:02.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3399927808 bytes 2024-09-16T23:09:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3439072256 bytes 2024-09-16T23:09:02.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3468617728 bytes 2024-09-16T23:09:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3504572416 bytes 2024-09-16T23:09:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3558152192 bytes 2024-09-16T23:09:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3615106048 bytes 2024-09-16T23:09:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3642636288 bytes 2024-09-16T23:09:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3680855040 bytes 2024-09-16T23:09:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3689424896 bytes 2024-09-16T23:09:03.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3690052608 bytes 2024-09-16T23:09:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3724871680 bytes 2024-09-16T23:09:03.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3751180288 bytes 2024-09-16T23:09:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3781824512 bytes 2024-09-16T23:09:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3883147264 bytes 2024-09-16T23:09:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3973495808 bytes 2024-09-16T23:09:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3984151552 bytes 2024-09-16T23:09:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4045284352 bytes 2024-09-16T23:09:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4062487552 bytes 2024-09-16T23:09:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4099712000 bytes 2024-09-16T23:09:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4127015936 bytes 2024-09-16T23:09:04.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4132448256 bytes 2024-09-16T23:09:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4201718784 bytes 2024-09-16T23:09:04.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4202335232 bytes 2024-09-16T23:09:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4287481856 bytes 2024-09-16T23:09:04.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4293465088 bytes 2024-09-16T23:09:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4345824256 bytes 2024-09-16T23:09:05.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4395145216 bytes 2024-09-16T23:09:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4484685824 bytes 2024-09-16T23:09:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4504655872 bytes 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:09:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:09:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:09:06.313 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.34854394996719096) > weight (0.25) 2024-09-16T23:09:06.314 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.30367320615015037) > weight (0.25) 2024-09-16T23:09:06.314 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T23:09:06.314 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T23:09:06.315 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-16T23:09:06.315 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:09:06.318+0000 7f0983049640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 36920) UID: 1000 2024-09-16T23:09:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:09:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:09:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:09:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:09:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":17894058.67,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"} 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:09:00 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:09:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:09:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:08:58' 2024-09-16T23:09:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:09:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:09:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:09:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":17894058.67,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"} 2024-09-16T23:09:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:09:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:09:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:09:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:09:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:09:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:09:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:09:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:09:00' 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:09:00' 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:09:00' 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:09:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:09:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='UNKNOWNUNKNOWNOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK4220mirror.2smithi09819.3.0-4883-g6e8474a0falseUNKNOWN42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":17894058.67,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:09:00cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:08:58' 2024-09-16T23:09:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:09:06.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:09:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:09:00' 2024-09-16T23:09:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:09:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:09:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap2 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap2 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2 2024-09-16T23:09:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:09:10.316 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T23:09:10.316 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:09:10.316 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T23:09:10.316 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T23:09:10.317 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T23:09:10.320 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T23:09:10.321 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-16T23:09:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:09:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:09:36.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:09:36.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:09:36.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:09:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:09:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:09:36.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:09:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:09:36.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:09:36.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:09:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:09:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:09:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:09:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:09:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:09:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:09:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:09:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:09:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:09:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:09:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:09:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:09:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:09:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:09:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:09:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:09:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:09:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:09:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:09:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:09:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:09:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 8 2024-09-16T23:09:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:09:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:38.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:09:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:09:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:09:38.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:09:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=8 2024-09-16T23:09:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:09:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:09:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:09:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:09:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:09:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:09:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:09:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:09:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:09:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:09:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 8 = '' 2024-09-16T23:09:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:09:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:09:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:09:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:09:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:09:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:09:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:09:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:09:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:09:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 8 = '' 2024-09-16T23:09:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:09:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:09:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:09:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:09:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:09:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:09:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:09:40.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:09:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:09:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 8 = '' 2024-09-16T23:09:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:09:40.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:09:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:41.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:09:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:09:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:09:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:09:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=8 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 8 = 8 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:09:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:09:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-16T23:09:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:09:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:09:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:09:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 4 2024-09-16T23:09:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:09:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:09:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap1-clean ']' 2024-09-16T23:09:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1-clean 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:09:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:09:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:09:43.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:09:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:09:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:09:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:09:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap2-clean 2024-09-16T23:09:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-16T23:09:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2-clean 2024-09-16T23:09:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:09:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:09:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:09:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:09:45.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:09:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:09:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:09:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:09:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 19587072 bytes 2024-09-16T23:09:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 82680832 bytes 2024-09-16T23:09:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 138198016 bytes 2024-09-16T23:09:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 162829312 bytes 2024-09-16T23:09:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 194611200 bytes 2024-09-16T23:09:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 254353408 bytes 2024-09-16T23:09:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 277965824 bytes 2024-09-16T23:09:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 366118912 bytes 2024-09-16T23:09:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 386601984 bytes 2024-09-16T23:09:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 459504640 bytes 2024-09-16T23:09:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 486202368 bytes 2024-09-16T23:09:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 515945472 bytes 2024-09-16T23:09:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 556875776 bytes 2024-09-16T23:09:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 585663488 bytes 2024-09-16T23:09:47.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2508074311514914) > weight (0.25) 2024-09-16T23:09:47.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7242460281743539) > weight (0.25) 2024-09-16T23:09:47.316 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9901162869453187) > weight (0.25) 2024-09-16T23:09:47.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6564631506149471) > weight (0.25) 2024-09-16T23:09:47.317 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-16T23:09:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 653112320 bytes 2024-09-16T23:09:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 653720576 bytes 2024-09-16T23:09:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 656315392 bytes 2024-09-16T23:09:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 681482240 bytes 2024-09-16T23:09:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 763569152 bytes 2024-09-16T23:09:47.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 794596352 bytes 2024-09-16T23:09:47.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 863797248 bytes 2024-09-16T23:09:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 934574080 bytes 2024-09-16T23:09:48.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1004549120 bytes 2024-09-16T23:09:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1112171520 bytes 2024-09-16T23:09:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1197685760 bytes 2024-09-16T23:09:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1278787584 bytes 2024-09-16T23:09:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1327374336 bytes 2024-09-16T23:09:48.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1359920128 bytes 2024-09-16T23:09:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1398140928 bytes 2024-09-16T23:09:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1447832576 bytes 2024-09-16T23:09:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1456045056 bytes 2024-09-16T23:09:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1492409344 bytes 2024-09-16T23:09:49.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1552940032 bytes 2024-09-16T23:09:49.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1590626304 bytes 2024-09-16T23:09:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1678624768 bytes 2024-09-16T23:09:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1745570816 bytes 2024-09-16T23:09:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1821965312 bytes 2024-09-16T23:09:49.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1824345088 bytes 2024-09-16T23:09:49.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1864263680 bytes 2024-09-16T23:09:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1907406848 bytes 2024-09-16T23:09:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1985388544 bytes 2024-09-16T23:09:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2003453952 bytes 2024-09-16T23:09:50.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2025063424 bytes 2024-09-16T23:09:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2069477376 bytes 2024-09-16T23:09:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2092761088 bytes 2024-09-16T23:09:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2152120320 bytes 2024-09-16T23:09:50.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2200566784 bytes 2024-09-16T23:09:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2284250112 bytes 2024-09-16T23:09:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2356390912 bytes 2024-09-16T23:09:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2395772928 bytes 2024-09-16T23:09:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2416960512 bytes 2024-09-16T23:09:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2454313984 bytes 2024-09-16T23:09:51.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2474000384 bytes 2024-09-16T23:09:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2476914688 bytes 2024-09-16T23:09:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2580552704 bytes 2024-09-16T23:09:51.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2663270400 bytes 2024-09-16T23:09:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2747294720 bytes 2024-09-16T23:09:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2761113600 bytes 2024-09-16T23:09:51.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2769220608 bytes 2024-09-16T23:09:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2792088576 bytes 2024-09-16T23:09:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2834548736 bytes 2024-09-16T23:09:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2835181568 bytes 2024-09-16T23:09:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2838654976 bytes 2024-09-16T23:09:51.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2878301184 bytes 2024-09-16T23:09:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2934743040 bytes 2024-09-16T23:09:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2998332416 bytes 2024-09-16T23:09:52.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3057825792 bytes 2024-09-16T23:09:52.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3099713536 bytes 2024-09-16T23:09:52.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3133389824 bytes 2024-09-16T23:09:52.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3236578304 bytes 2024-09-16T23:09:52.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3239820288 bytes 2024-09-16T23:09:52.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3274102784 bytes 2024-09-16T23:09:52.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3295389696 bytes 2024-09-16T23:09:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3300630528 bytes 2024-09-16T23:09:52.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3302552576 bytes 2024-09-16T23:09:53.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3354532864 bytes 2024-09-16T23:09:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3355154432 bytes 2024-09-16T23:09:53.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3371584512 bytes 2024-09-16T23:09:53.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3418220544 bytes 2024-09-16T23:09:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3487876096 bytes 2024-09-16T23:09:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3488500736 bytes 2024-09-16T23:09:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3505716224 bytes 2024-09-16T23:09:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3550017536 bytes 2024-09-16T23:09:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3611245568 bytes 2024-09-16T23:09:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3664402432 bytes 2024-09-16T23:09:53.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3725981696 bytes 2024-09-16T23:09:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3776216064 bytes 2024-09-16T23:09:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3800176640 bytes 2024-09-16T23:09:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3855787008 bytes 2024-09-16T23:09:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3913261056 bytes 2024-09-16T23:09:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3931838464 bytes 2024-09-16T23:09:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3973548032 bytes 2024-09-16T23:09:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4013546496 bytes 2024-09-16T23:09:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4050391040 bytes 2024-09-16T23:09:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4056312832 bytes 2024-09-16T23:09:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4063457280 bytes 2024-09-16T23:09:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4096906240 bytes 2024-09-16T23:09:54.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4122952704 bytes 2024-09-16T23:09:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4177061888 bytes 2024-09-16T23:09:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4247785472 bytes 2024-09-16T23:09:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4270625792 bytes 2024-09-16T23:09:55.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4314836992 bytes 2024-09-16T23:09:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4350101504 bytes 2024-09-16T23:09:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4431791104 bytes 2024-09-16T23:09:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4465725440 bytes 2024-09-16T23:09:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4562622464 bytes 2024-09-16T23:09:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4615086080 bytes 2024-09-16T23:09:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4649549824 bytes 2024-09-16T23:09:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4670620672 bytes 2024-09-16T23:09:56.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4681769984 bytes 2024-09-16T23:09:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4727395328 bytes 2024-09-16T23:09:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4821699584 bytes 2024-09-16T23:09:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4844625920 bytes 2024-09-16T23:09:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4871706624 bytes 2024-09-16T23:09:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 4892277760 bytes 2024-09-16T23:09:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 4913615872 bytes 2024-09-16T23:09:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 4946093056 bytes 2024-09-16T23:09:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 5008569344 bytes 2024-09-16T23:09:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 5023259648 bytes 2024-09-16T23:09:57.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 5113561088 bytes 2024-09-16T23:09:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5193379840 bytes 2024-09-16T23:09:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 5220954112 bytes 2024-09-16T23:09:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12300 IOs, 5244706816 bytes 2024-09-16T23:09:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12400 IOs, 5263196160 bytes 2024-09-16T23:09:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12500 IOs, 5303028736 bytes 2024-09-16T23:09:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12600 IOs, 5303641088 bytes 2024-09-16T23:09:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12700 IOs, 5327795200 bytes 2024-09-16T23:09:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12800 IOs, 5368709120 bytes 2024-09-16T23:09:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12900 IOs, 5388786688 bytes 2024-09-16T23:09:57.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13000 IOs, 5422158848 bytes 2024-09-16T23:09:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13100 IOs, 5464372224 bytes 2024-09-16T23:09:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13200 IOs, 5535183872 bytes 2024-09-16T23:09:58.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13300 IOs, 5564406784 bytes 2024-09-16T23:09:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13400 IOs, 5625730048 bytes 2024-09-16T23:09:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13500 IOs, 5718231040 bytes 2024-09-16T23:09:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13600 IOs, 5731715072 bytes 2024-09-16T23:09:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13700 IOs, 5776281600 bytes 2024-09-16T23:09:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13800 IOs, 5794485248 bytes 2024-09-16T23:09:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13900 IOs, 5874696192 bytes 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:09:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:09:58.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:09:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:09:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:09:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:09:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"} 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:09:30 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:09:58' 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:09:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:09:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:09:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"} 2024-09-16T23:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:09:59.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:09:30' 2024-09-16T23:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:09:30' 2024-09-16T23:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:09:30' 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:09:59.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:10:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK43714374mirror.2smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268410880.0,"last_snapshot_bytes":536821760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528130,"remote_snapshot_timestamp":1726528130,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:09:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:09:58' 2024-09-16T23:10:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:10:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:10:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:09:30' 2024-09-16T23:10:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:10:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:10:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:10:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:10:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap3 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap3 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3 2024-09-16T23:10:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:10:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:10:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:10:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:10:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:10:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:10:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:10:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:10:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:10:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:10:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:10:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:10:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:10:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:10:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:10:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:10:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:10:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:10:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:10:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:10:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:10:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:10:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:10:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:10:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:10:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:10:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:10:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:10:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:10:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:10:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:10:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:10:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:10:30.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 11 2024-09-16T23:10:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:10:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:10:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:10:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:10:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:10:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=11 2024-09-16T23:10:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:10:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:10:30.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:10:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:10:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:10:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:10:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 11 = '' 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:10:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:10:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:10:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:10:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:10:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 11 = '' 2024-09-16T23:10:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:10:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:10:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:10:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:10:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 11 = '' 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:10:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:10:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:10:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:10:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=11 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 11 = 11 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3 2024-09-16T23:10:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:10:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:10:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:10:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-16T23:10:34.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 6 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:09:43 2024 2024-09-16T23:10:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:10:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:10:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:10:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:10:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:10:34.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap2-clean ']' 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2-clean 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:34.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:10:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:10:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:10:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T23:10:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3 2024-09-16T23:10:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:10:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:10:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:10:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:10:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:37.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:10:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:10:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap3-clean 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3-clean 2024-09-16T23:10:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:10:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:10:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:10:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:10:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:10:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:10:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:10:38.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 11685888 bytes 2024-09-16T23:10:38.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 49458176 bytes 2024-09-16T23:10:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 94554112 bytes 2024-09-16T23:10:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 141437952 bytes 2024-09-16T23:10:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 194874368 bytes 2024-09-16T23:10:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 267494400 bytes 2024-09-16T23:10:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 310916096 bytes 2024-09-16T23:10:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 376591360 bytes 2024-09-16T23:10:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 397591552 bytes 2024-09-16T23:10:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 449667072 bytes 2024-09-16T23:10:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 519953408 bytes 2024-09-16T23:10:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 583197696 bytes 2024-09-16T23:10:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 694444032 bytes 2024-09-16T23:10:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 715679744 bytes 2024-09-16T23:10:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 751899648 bytes 2024-09-16T23:10:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 788978688 bytes 2024-09-16T23:10:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 881215488 bytes 2024-09-16T23:10:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 888205312 bytes 2024-09-16T23:10:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 946398208 bytes 2024-09-16T23:10:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1021409280 bytes 2024-09-16T23:10:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1091209216 bytes 2024-09-16T23:10:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1156090880 bytes 2024-09-16T23:10:40.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1218459648 bytes 2024-09-16T23:10:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1244512256 bytes 2024-09-16T23:10:41.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1292513280 bytes 2024-09-16T23:10:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1323880448 bytes 2024-09-16T23:10:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1341524992 bytes 2024-09-16T23:10:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1418847232 bytes 2024-09-16T23:10:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1504712704 bytes 2024-09-16T23:10:41.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1565988864 bytes 2024-09-16T23:10:41.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1583384576 bytes 2024-09-16T23:10:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1626859520 bytes 2024-09-16T23:10:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1738128384 bytes 2024-09-16T23:10:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1766077440 bytes 2024-09-16T23:10:42.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1820690432 bytes 2024-09-16T23:10:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1857480704 bytes 2024-09-16T23:10:42.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1899108352 bytes 2024-09-16T23:10:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1937913856 bytes 2024-09-16T23:10:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 2040223744 bytes 2024-09-16T23:10:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 2108222464 bytes 2024-09-16T23:10:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 2144491520 bytes 2024-09-16T23:10:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2214393856 bytes 2024-09-16T23:10:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2248306688 bytes 2024-09-16T23:10:43.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2295094272 bytes 2024-09-16T23:10:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2336403456 bytes 2024-09-16T23:10:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2380850176 bytes 2024-09-16T23:10:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2442813440 bytes 2024-09-16T23:10:43.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2512088064 bytes 2024-09-16T23:10:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2558583808 bytes 2024-09-16T23:10:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2573464576 bytes 2024-09-16T23:10:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2605457408 bytes 2024-09-16T23:10:43.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2653785088 bytes 2024-09-16T23:10:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2666298368 bytes 2024-09-16T23:10:44.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2711368704 bytes 2024-09-16T23:10:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2725141504 bytes 2024-09-16T23:10:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2756513792 bytes 2024-09-16T23:10:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2792682496 bytes 2024-09-16T23:10:44.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2861537280 bytes 2024-09-16T23:10:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2887585792 bytes 2024-09-16T23:10:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2973827072 bytes 2024-09-16T23:10:44.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 3036635136 bytes 2024-09-16T23:10:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 3089441792 bytes 2024-09-16T23:10:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 3196465152 bytes 2024-09-16T23:10:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3224248320 bytes 2024-09-16T23:10:45.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3244291072 bytes 2024-09-16T23:10:45.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3269709824 bytes 2024-09-16T23:10:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3289804800 bytes 2024-09-16T23:10:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3362495488 bytes 2024-09-16T23:10:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3413971968 bytes 2024-09-16T23:10:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3415228416 bytes 2024-09-16T23:10:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3440185344 bytes 2024-09-16T23:10:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3479244800 bytes 2024-09-16T23:10:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3543648256 bytes 2024-09-16T23:10:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3602608128 bytes 2024-09-16T23:10:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3634805760 bytes 2024-09-16T23:10:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3689530368 bytes 2024-09-16T23:10:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3699789824 bytes 2024-09-16T23:10:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3718129664 bytes 2024-09-16T23:10:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3773799424 bytes 2024-09-16T23:10:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3845194752 bytes 2024-09-16T23:10:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3845818368 bytes 2024-09-16T23:10:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3909673984 bytes 2024-09-16T23:10:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3938798592 bytes 2024-09-16T23:10:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 4000565248 bytes 2024-09-16T23:10:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 4057659392 bytes 2024-09-16T23:10:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 4096819200 bytes 2024-09-16T23:10:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 4130267136 bytes 2024-09-16T23:10:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 4165854208 bytes 2024-09-16T23:10:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 4209176576 bytes 2024-09-16T23:10:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4261641216 bytes 2024-09-16T23:10:47.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4316223488 bytes 2024-09-16T23:10:47.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4324631552 bytes 2024-09-16T23:10:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4382105600 bytes 2024-09-16T23:10:48.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4471061504 bytes 2024-09-16T23:10:48.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4492051456 bytes 2024-09-16T23:10:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4527227904 bytes 2024-09-16T23:10:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4570661888 bytes 2024-09-16T23:10:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4612370432 bytes 2024-09-16T23:10:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4678633472 bytes 2024-09-16T23:10:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4717382656 bytes 2024-09-16T23:10:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4747681792 bytes 2024-09-16T23:10:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4801988608 bytes 2024-09-16T23:10:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4837158912 bytes 2024-09-16T23:10:49.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4864318464 bytes 2024-09-16T23:10:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4875845632 bytes 2024-09-16T23:10:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4876448768 bytes 2024-09-16T23:10:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 5035493376 bytes 2024-09-16T23:10:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 5070995456 bytes 2024-09-16T23:10:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 5129798656 bytes 2024-09-16T23:10:49.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 5197414400 bytes 2024-09-16T23:10:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 5241613312 bytes 2024-09-16T23:10:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 5303011328 bytes 2024-09-16T23:10:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 5346890752 bytes 2024-09-16T23:10:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 5422245888 bytes 2024-09-16T23:10:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 5445915648 bytes 2024-09-16T23:10:50.321 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.33487664525618677) > weight (0.25) 2024-09-16T23:10:50.322 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8582991091462707) > weight (0.25) 2024-09-16T23:10:50.322 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T23:10:50.322 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T23:10:50.322 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T23:10:50.323 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T23:10:50.323 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:10:50.323 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:10:50.326+0000 7f5f2a10f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 39474) UID: 1000 2024-09-16T23:10:50.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 5501474816 bytes 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:10:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:10:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:10:50.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:10:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:10:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:10:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:10:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534596096.0,"last_snapshot_bytes":532370432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528177,"remote_snapshot_timestamp":1726528177,"replay_state":"idle"} 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:10:30 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:10:28' 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:10:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:10:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:10:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534596096.0,"last_snapshot_bytes":532370432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528177,"remote_snapshot_timestamp":1726528177,"replay_state":"idle"} 2024-09-16T23:10:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:10:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:10:51.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:10:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:10:51.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:10:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:10:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:10:51.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:10:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:10:30' 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:10:30' 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:10:30' 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:10:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:10:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534596096.0,"last_snapshot_bytes":532370432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528177,"remote_snapshot_timestamp":1726528177,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:10:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:10:28' 2024-09-16T23:10:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:10:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:10:51.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:10:30' 2024-09-16T23:10:51.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:10:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:10:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap4 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap4 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4 2024-09-16T23:10:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:10:56.425 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T23:10:56.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T23:10:56.426 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T23:10:56.429 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T23:10:56.429 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-16T23:11:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:11:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:11:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:11:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:11:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:11:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:11:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:11:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:11:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:11:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:11:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:11:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:11:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:11:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:11:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:11:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:11:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:11:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:11:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:11:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:11:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:11:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:11:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:11:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:11:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:11:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:11:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:11:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:11:20.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:11:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:11:20.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:11:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:11:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:11:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:11:20.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:11:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:11:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:11:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:11:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:11:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:11:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:11:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:11:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:11:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:11:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:11:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:11:20.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:11:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 14 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:11:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:11:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:11:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:11:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=14 2024-09-16T23:11:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:11:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:11:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:11:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:22.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:11:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:11:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:11:22.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:11:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 14 = '' 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:11:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:11:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:11:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:11:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:11:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:11:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:11:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:11:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:11:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 14 = '' 2024-09-16T23:11:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:11:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:11:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:11:24.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:11:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=14 2024-09-16T23:11:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 14 = 14 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:11:24.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:11:25.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-16T23:11:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:11:25.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 8 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:09:43 2024 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:11:25.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:11:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:11:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap3-clean ']' 2024-09-16T23:11:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3-clean 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:11:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:11:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:26.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8389071260660079) > weight (0.25) 2024-09-16T23:11:26.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5483426957742276) > weight (0.25) 2024-09-16T23:11:26.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8353658503980252) > weight (0.25) 2024-09-16T23:11:26.318 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T23:11:26.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T23:11:26.319 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T23:11:26.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:11:26.322+0000 7f82a7026640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 37012) UID: 1000 2024-09-16T23:11:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:11:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:11:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:11:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:11:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:11:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:28.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:11:28.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap4-clean 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4-clean 2024-09-16T23:11:28.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:11:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:11:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:11:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:11:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:11:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:11:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:11:28.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 73141248 bytes 2024-09-16T23:11:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 98696192 bytes 2024-09-16T23:11:28.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 231129088 bytes 2024-09-16T23:11:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 234969088 bytes 2024-09-16T23:11:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 300488704 bytes 2024-09-16T23:11:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 340548608 bytes 2024-09-16T23:11:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 351766528 bytes 2024-09-16T23:11:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 384664576 bytes 2024-09-16T23:11:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 413359104 bytes 2024-09-16T23:11:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 505586688 bytes 2024-09-16T23:11:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 593566720 bytes 2024-09-16T23:11:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 625564672 bytes 2024-09-16T23:11:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 638278656 bytes 2024-09-16T23:11:30.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 686794752 bytes 2024-09-16T23:11:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 748908544 bytes 2024-09-16T23:11:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 778149888 bytes 2024-09-16T23:11:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 823274496 bytes 2024-09-16T23:11:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 844893184 bytes 2024-09-16T23:11:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 845828096 bytes 2024-09-16T23:11:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 912568320 bytes 2024-09-16T23:11:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 959764480 bytes 2024-09-16T23:11:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 960361472 bytes 2024-09-16T23:11:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 979658752 bytes 2024-09-16T23:11:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1021878272 bytes 2024-09-16T23:11:31.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1052889088 bytes 2024-09-16T23:11:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1113595904 bytes 2024-09-16T23:11:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1123500032 bytes 2024-09-16T23:11:31.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1150104576 bytes 2024-09-16T23:11:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1201024000 bytes 2024-09-16T23:11:31.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1327790080 bytes 2024-09-16T23:11:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1364220928 bytes 2024-09-16T23:11:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1389806592 bytes 2024-09-16T23:11:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1393665024 bytes 2024-09-16T23:11:31.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1451906048 bytes 2024-09-16T23:11:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1452513280 bytes 2024-09-16T23:11:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1482291200 bytes 2024-09-16T23:11:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1510296576 bytes 2024-09-16T23:11:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1527493632 bytes 2024-09-16T23:11:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1559318528 bytes 2024-09-16T23:11:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1595565056 bytes 2024-09-16T23:11:32.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1687786496 bytes 2024-09-16T23:11:32.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1733989376 bytes 2024-09-16T23:11:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1848396800 bytes 2024-09-16T23:11:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1869187072 bytes 2024-09-16T23:11:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1902134272 bytes 2024-09-16T23:11:33.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1958638592 bytes 2024-09-16T23:11:33.320 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T23:11:33.320 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:11:33.320 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T23:11:33.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T23:11:33.320 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T23:11:33.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T23:11:33.324 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2024-09-16T23:11:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1983672320 bytes 2024-09-16T23:11:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2011654144 bytes 2024-09-16T23:11:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2033822720 bytes 2024-09-16T23:11:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2071655424 bytes 2024-09-16T23:11:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2110693376 bytes 2024-09-16T23:11:33.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2135064576 bytes 2024-09-16T23:11:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2199096320 bytes 2024-09-16T23:11:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2253350912 bytes 2024-09-16T23:11:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2278412288 bytes 2024-09-16T23:11:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2289340416 bytes 2024-09-16T23:11:34.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2349805568 bytes 2024-09-16T23:11:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2434650112 bytes 2024-09-16T23:11:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2454785024 bytes 2024-09-16T23:11:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2471232512 bytes 2024-09-16T23:11:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2557131776 bytes 2024-09-16T23:11:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2655199232 bytes 2024-09-16T23:11:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2699667456 bytes 2024-09-16T23:11:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2739283968 bytes 2024-09-16T23:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2772562944 bytes 2024-09-16T23:11:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2842474496 bytes 2024-09-16T23:11:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2908902400 bytes 2024-09-16T23:11:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2909528064 bytes 2024-09-16T23:11:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2932966400 bytes 2024-09-16T23:11:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2961201152 bytes 2024-09-16T23:11:35.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3018611712 bytes 2024-09-16T23:11:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3040337920 bytes 2024-09-16T23:11:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3083669504 bytes 2024-09-16T23:11:36.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3163642880 bytes 2024-09-16T23:11:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3220123648 bytes 2024-09-16T23:11:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3245819904 bytes 2024-09-16T23:11:36.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3250159616 bytes 2024-09-16T23:11:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3303522304 bytes 2024-09-16T23:11:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3332314112 bytes 2024-09-16T23:11:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3381550080 bytes 2024-09-16T23:11:36.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3416507392 bytes 2024-09-16T23:11:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3476514816 bytes 2024-09-16T23:11:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3543864320 bytes 2024-09-16T23:11:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3544463360 bytes 2024-09-16T23:11:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3615472640 bytes 2024-09-16T23:11:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3654003712 bytes 2024-09-16T23:11:37.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3722182656 bytes 2024-09-16T23:11:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3804731392 bytes 2024-09-16T23:11:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3823783936 bytes 2024-09-16T23:11:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3860685824 bytes 2024-09-16T23:11:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3918452736 bytes 2024-09-16T23:11:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3961923584 bytes 2024-09-16T23:11:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3985605632 bytes 2024-09-16T23:11:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4028600320 bytes 2024-09-16T23:11:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4105448448 bytes 2024-09-16T23:11:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4137011200 bytes 2024-09-16T23:11:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4218883072 bytes 2024-09-16T23:11:38.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4304947200 bytes 2024-09-16T23:11:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4357657600 bytes 2024-09-16T23:11:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4368398336 bytes 2024-09-16T23:11:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4439217152 bytes 2024-09-16T23:11:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4451177472 bytes 2024-09-16T23:11:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4500349952 bytes 2024-09-16T23:11:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4572699648 bytes 2024-09-16T23:11:39.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4596450304 bytes 2024-09-16T23:11:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4613537792 bytes 2024-09-16T23:11:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4645838848 bytes 2024-09-16T23:11:39.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4707334144 bytes 2024-09-16T23:11:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4754502656 bytes 2024-09-16T23:11:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4799367168 bytes 2024-09-16T23:11:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4832842752 bytes 2024-09-16T23:11:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4883551232 bytes 2024-09-16T23:11:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4974305280 bytes 2024-09-16T23:11:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 5019889664 bytes 2024-09-16T23:11:40.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 5097318400 bytes 2024-09-16T23:11:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 5148626944 bytes 2024-09-16T23:11:40.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 5179721728 bytes 2024-09-16T23:11:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 5261999104 bytes 2024-09-16T23:11:40.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 5271231488 bytes 2024-09-16T23:11:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 5306627072 bytes 2024-09-16T23:11:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5327691776 bytes 2024-09-16T23:11:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 5357969408 bytes 2024-09-16T23:11:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12300 IOs, 5424031744 bytes 2024-09-16T23:11:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12400 IOs, 5471202304 bytes 2024-09-16T23:11:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12500 IOs, 5481838592 bytes 2024-09-16T23:11:41.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12600 IOs, 5495299072 bytes 2024-09-16T23:11:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:11:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:11:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:11:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:11:42.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":17879517.87,"bytes_per_snapshot":536368128.0,"last_snapshot_bytes":536385536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528281,"remote_snapshot_timestamp":1726528281,"replay_state":"idle"} 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:11:30 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:11:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:11:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:11:28' 2024-09-16T23:11:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:11:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:11:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:11:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":17879517.87,"bytes_per_snapshot":536368128.0,"last_snapshot_bytes":536385536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528281,"remote_snapshot_timestamp":1726528281,"replay_state":"idle"} 2024-09-16T23:11:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:11:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:11:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:11:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:11:42.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:11:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:11:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:11:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:11:30' 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:11:30' 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:11:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:11:30' 2024-09-16T23:11:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:11:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:11:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:11:42.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":17879517.87,"bytes_per_snapshot":536368128.0,"last_snapshot_bytes":536385536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528281,"remote_snapshot_timestamp":1726528281,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:11:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:11:28' 2024-09-16T23:11:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:11:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:11:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:11:30' 2024-09-16T23:11:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:11:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:11:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:11:42.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:11:42.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap5 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap5 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:11:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5 2024-09-16T23:11:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:12:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:12:10.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:12:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:12:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:12:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:12:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:12:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:12:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:12:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:12:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:12:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:12:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:12:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:12:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:12:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:12:11.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:12:11.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:12:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:12:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:12:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:12:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:12:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:12:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:12:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:12:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:12:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:12:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:12:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:12:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:12:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:12:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:12:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:12:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 17 2024-09-16T23:12:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:12:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:12:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:12:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:12:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=17 2024-09-16T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:12:13.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:12:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:12:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:12:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:12:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:12:13.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:12:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:12:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:12:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 17 = '' 2024-09-16T23:12:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:12:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:12:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:12:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:12:14.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:12:14.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:12:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:12:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:12:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:12:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:12:14.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 17 = '' 2024-09-16T23:12:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:12:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:12:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:12:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:12:15.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:12:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 17 = '' 2024-09-16T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:12:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:12:16.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:12:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=17 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 17 = 17 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:12:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:12:17.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 10 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:12:17.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:11:25 2024 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:12:17.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:12:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap4-clean ']' 2024-09-16T23:12:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T23:12:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4-clean 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:12:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:12:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:12:18.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:12:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:12:20.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap5-clean 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5-clean 2024-09-16T23:12:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:12:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:12:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:12:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:12:22.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 30915584 bytes 2024-09-16T23:12:22.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 92693504 bytes 2024-09-16T23:12:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 114978816 bytes 2024-09-16T23:12:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 158170112 bytes 2024-09-16T23:12:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 184714240 bytes 2024-09-16T23:12:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 248704000 bytes 2024-09-16T23:12:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 255807488 bytes 2024-09-16T23:12:22.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 309361664 bytes 2024-09-16T23:12:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 337190912 bytes 2024-09-16T23:12:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 360894464 bytes 2024-09-16T23:12:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 365806592 bytes 2024-09-16T23:12:23.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 373426176 bytes 2024-09-16T23:12:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 411569152 bytes 2024-09-16T23:12:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 444631040 bytes 2024-09-16T23:12:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 478751744 bytes 2024-09-16T23:12:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 503239680 bytes 2024-09-16T23:12:23.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 589161472 bytes 2024-09-16T23:12:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 665353216 bytes 2024-09-16T23:12:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 689940480 bytes 2024-09-16T23:12:23.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 757979136 bytes 2024-09-16T23:12:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 844994560 bytes 2024-09-16T23:12:24.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 864098304 bytes 2024-09-16T23:12:24.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 949621760 bytes 2024-09-16T23:12:24.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 973209600 bytes 2024-09-16T23:12:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1034438656 bytes 2024-09-16T23:12:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1079054336 bytes 2024-09-16T23:12:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1099708416 bytes 2024-09-16T23:12:24.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1136242688 bytes 2024-09-16T23:12:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1171819520 bytes 2024-09-16T23:12:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1186067456 bytes 2024-09-16T23:12:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1228910592 bytes 2024-09-16T23:12:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1357171712 bytes 2024-09-16T23:12:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1451397120 bytes 2024-09-16T23:12:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1530404864 bytes 2024-09-16T23:12:25.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1548754944 bytes 2024-09-16T23:12:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1585272832 bytes 2024-09-16T23:12:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1631879168 bytes 2024-09-16T23:12:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1677721600 bytes 2024-09-16T23:12:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1775207424 bytes 2024-09-16T23:12:26.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1832182784 bytes 2024-09-16T23:12:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1883482112 bytes 2024-09-16T23:12:26.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1969989632 bytes 2024-09-16T23:12:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1991199744 bytes 2024-09-16T23:12:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2103211008 bytes 2024-09-16T23:12:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2131594240 bytes 2024-09-16T23:12:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2183293952 bytes 2024-09-16T23:12:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2263448576 bytes 2024-09-16T23:12:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2273505280 bytes 2024-09-16T23:12:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2300533760 bytes 2024-09-16T23:12:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2396612608 bytes 2024-09-16T23:12:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2431025152 bytes 2024-09-16T23:12:27.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2462222336 bytes 2024-09-16T23:12:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2534566912 bytes 2024-09-16T23:12:27.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2547914752 bytes 2024-09-16T23:12:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2571152384 bytes 2024-09-16T23:12:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2622047232 bytes 2024-09-16T23:12:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2652480512 bytes 2024-09-16T23:12:28.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2713290752 bytes 2024-09-16T23:12:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2785568768 bytes 2024-09-16T23:12:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2867212288 bytes 2024-09-16T23:12:28.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2894425088 bytes 2024-09-16T23:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2954461184 bytes 2024-09-16T23:12:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2996923392 bytes 2024-09-16T23:12:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3071316992 bytes 2024-09-16T23:12:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3135043584 bytes 2024-09-16T23:12:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3178141696 bytes 2024-09-16T23:12:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3207483392 bytes 2024-09-16T23:12:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3268655104 bytes 2024-09-16T23:12:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3301198848 bytes 2024-09-16T23:12:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3352298496 bytes 2024-09-16T23:12:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3390944256 bytes 2024-09-16T23:12:29.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3418058752 bytes 2024-09-16T23:12:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3454324736 bytes 2024-09-16T23:12:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3486739456 bytes 2024-09-16T23:12:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3508629504 bytes 2024-09-16T23:12:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3587518464 bytes 2024-09-16T23:12:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3647070208 bytes 2024-09-16T23:12:30.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3664372736 bytes 2024-09-16T23:12:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3699927040 bytes 2024-09-16T23:12:30.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3753481216 bytes 2024-09-16T23:12:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3760845824 bytes 2024-09-16T23:12:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3792102400 bytes 2024-09-16T23:12:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3816135680 bytes 2024-09-16T23:12:30.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3895891968 bytes 2024-09-16T23:12:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3944836096 bytes 2024-09-16T23:12:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3945462784 bytes 2024-09-16T23:12:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3995057152 bytes 2024-09-16T23:12:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 4080807936 bytes 2024-09-16T23:12:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 4129118208 bytes 2024-09-16T23:12:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4162634752 bytes 2024-09-16T23:12:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4186240000 bytes 2024-09-16T23:12:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4262323200 bytes 2024-09-16T23:12:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4267069440 bytes 2024-09-16T23:12:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4289803264 bytes 2024-09-16T23:12:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4327166976 bytes 2024-09-16T23:12:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4350297088 bytes 2024-09-16T23:12:31.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4375645184 bytes 2024-09-16T23:12:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4434900992 bytes 2024-09-16T23:12:32.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4456770560 bytes 2024-09-16T23:12:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4497098752 bytes 2024-09-16T23:12:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4521217024 bytes 2024-09-16T23:12:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4602442752 bytes 2024-09-16T23:12:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4717483008 bytes 2024-09-16T23:12:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4775056384 bytes 2024-09-16T23:12:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4849431552 bytes 2024-09-16T23:12:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4868734976 bytes 2024-09-16T23:12:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4914771968 bytes 2024-09-16T23:12:33.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4947103744 bytes 2024-09-16T23:12:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 5000161280 bytes 2024-09-16T23:12:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 5045155840 bytes 2024-09-16T23:12:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 5068408832 bytes 2024-09-16T23:12:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 5089469440 bytes 2024-09-16T23:12:33.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 5177013248 bytes 2024-09-16T23:12:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 5264482304 bytes 2024-09-16T23:12:33.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 5299414016 bytes 2024-09-16T23:12:34.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 5323589632 bytes 2024-09-16T23:12:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 5396115456 bytes 2024-09-16T23:12:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 5463552000 bytes 2024-09-16T23:12:34.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 5494224896 bytes 2024-09-16T23:12:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 5531817984 bytes 2024-09-16T23:12:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5585148928 bytes 2024-09-16T23:12:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 5596599296 bytes 2024-09-16T23:12:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12300 IOs, 5630249984 bytes 2024-09-16T23:12:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12400 IOs, 5666254848 bytes 2024-09-16T23:12:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12500 IOs, 5743517696 bytes 2024-09-16T23:12:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12600 IOs, 5746831360 bytes 2024-09-16T23:12:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12700 IOs, 5821622272 bytes 2024-09-16T23:12:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12800 IOs, 5838995456 bytes 2024-09-16T23:12:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12900 IOs, 5867119616 bytes 2024-09-16T23:12:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13000 IOs, 5867728896 bytes 2024-09-16T23:12:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13100 IOs, 5915355136 bytes 2024-09-16T23:12:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13200 IOs, 5975550976 bytes 2024-09-16T23:12:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13300 IOs, 6009847808 bytes 2024-09-16T23:12:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:13400 IOs, 6060413952 bytes 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:12:35.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:12:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:12:35.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:12:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:12:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":17869414.4,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"} 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:12:30 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:12:36.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:12:28' 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:12:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:12:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":17869414.4,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"} 2024-09-16T23:12:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:12:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:12:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:12:36.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:12:36.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:12:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:12:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:12:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:12:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:12:30' 2024-09-16T23:12:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:12:30' 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:12:30' 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:12:36.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:12:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":17869414.4,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:12:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:12:28' 2024-09-16T23:12:37.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:12:37.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:12:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:12:30' 2024-09-16T23:12:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:12:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:12:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:12:37.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:12:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap6 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap6 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:12:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6 2024-09-16T23:12:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:12:52.430 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8128950322156389) > weight (0.25) 2024-09-16T23:12:52.430 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.500201371852564) > weight (0.25) 2024-09-16T23:12:52.430 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3909118171651256) > weight (0.25) 2024-09-16T23:12:52.430 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.7977797885275197) > weight (0.25) 2024-09-16T23:12:52.430 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T23:13:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:13:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:13:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:13:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:13:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:13:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:13:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:13:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:13:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:13:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:13:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:13:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:04.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:13:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:13:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:13:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:13:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:13:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:13:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:13:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:13:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:13:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:13:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:13:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:13:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:13:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:13:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:13:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:13:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:13:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:04.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:13:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:13:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:13:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:13:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:13:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:13:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:13:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:13:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:13:05.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5986707196948615) > weight (0.25) 2024-09-16T23:13:05.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6173902608147592) > weight (0.25) 2024-09-16T23:13:05.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.790474675185747) > weight (0.25) 2024-09-16T23:13:05.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9573543117311304) > weight (0.25) 2024-09-16T23:13:05.326 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2024-09-16T23:13:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 20 2024-09-16T23:13:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:13:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:13:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:13:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:13:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:13:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=20 2024-09-16T23:13:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:13:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:13:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:13:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:13:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:13:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:13:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:13:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:13:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:13:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:13:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 20 = '' 2024-09-16T23:13:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:13:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:13:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:13:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:13:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:13:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:13:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:13:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:13:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:13:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 20 = '' 2024-09-16T23:13:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:13:07.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:13:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:13:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:13:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:13:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:13:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:13:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:13:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:13:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 20 = '' 2024-09-16T23:13:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:13:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:13:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:13:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:13:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:13:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=20 2024-09-16T23:13:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 20 = 20 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:13:09.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:13:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-16T23:13:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:13:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 12 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:11:25 2024 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:13:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:13:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:13:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:13:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:13:10.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap5-clean ']' 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5-clean 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:13:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:13:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:13:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:13:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:13:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:13:13.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:13:13.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap6-clean 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6-clean 2024-09-16T23:13:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:13:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:13:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:13:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:13:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 8070144 bytes 2024-09-16T23:13:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 137136128 bytes 2024-09-16T23:13:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 167804928 bytes 2024-09-16T23:13:15.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 239875072 bytes 2024-09-16T23:13:15.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 250260480 bytes 2024-09-16T23:13:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 330359808 bytes 2024-09-16T23:13:15.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 357081088 bytes 2024-09-16T23:13:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 399165440 bytes 2024-09-16T23:13:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 460264448 bytes 2024-09-16T23:13:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 514754560 bytes 2024-09-16T23:13:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 539138048 bytes 2024-09-16T23:13:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 583562240 bytes 2024-09-16T23:13:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 611356672 bytes 2024-09-16T23:13:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 679872512 bytes 2024-09-16T23:13:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 746276864 bytes 2024-09-16T23:13:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 776327168 bytes 2024-09-16T23:13:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 854240256 bytes 2024-09-16T23:13:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 913343488 bytes 2024-09-16T23:13:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 975685632 bytes 2024-09-16T23:13:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1042148352 bytes 2024-09-16T23:13:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1071332352 bytes 2024-09-16T23:13:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1114356736 bytes 2024-09-16T23:13:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1143084032 bytes 2024-09-16T23:13:17.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1265203200 bytes 2024-09-16T23:13:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1273841664 bytes 2024-09-16T23:13:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1295767552 bytes 2024-09-16T23:13:17.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1313062912 bytes 2024-09-16T23:13:17.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1428713472 bytes 2024-09-16T23:13:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1453648896 bytes 2024-09-16T23:13:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1462984704 bytes 2024-09-16T23:13:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1519339520 bytes 2024-09-16T23:13:18.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1569841152 bytes 2024-09-16T23:13:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1652919296 bytes 2024-09-16T23:13:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1693266944 bytes 2024-09-16T23:13:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1716987904 bytes 2024-09-16T23:13:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1732626432 bytes 2024-09-16T23:13:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1787388928 bytes 2024-09-16T23:13:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1810454528 bytes 2024-09-16T23:13:18.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1881937920 bytes 2024-09-16T23:13:18.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1952141312 bytes 2024-09-16T23:13:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 2009929728 bytes 2024-09-16T23:13:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2097076224 bytes 2024-09-16T23:13:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2127968256 bytes 2024-09-16T23:13:19.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2203080704 bytes 2024-09-16T23:13:19.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2255881216 bytes 2024-09-16T23:13:19.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2331917312 bytes 2024-09-16T23:13:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2365006848 bytes 2024-09-16T23:13:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2415977472 bytes 2024-09-16T23:13:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2477465600 bytes 2024-09-16T23:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2501872640 bytes 2024-09-16T23:13:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2548385792 bytes 2024-09-16T23:13:20.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2606114816 bytes 2024-09-16T23:13:20.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2606755840 bytes 2024-09-16T23:13:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2638311424 bytes 2024-09-16T23:13:20.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2677259264 bytes 2024-09-16T23:13:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2711517184 bytes 2024-09-16T23:13:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2764275712 bytes 2024-09-16T23:13:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2772297728 bytes 2024-09-16T23:13:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2796342272 bytes 2024-09-16T23:13:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2838001664 bytes 2024-09-16T23:13:21.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2931441664 bytes 2024-09-16T23:13:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2953111552 bytes 2024-09-16T23:13:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 3000409088 bytes 2024-09-16T23:13:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3045167104 bytes 2024-09-16T23:13:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3045792768 bytes 2024-09-16T23:13:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3089655808 bytes 2024-09-16T23:13:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3120376832 bytes 2024-09-16T23:13:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3154265088 bytes 2024-09-16T23:13:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3193151488 bytes 2024-09-16T23:13:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3216695296 bytes 2024-09-16T23:13:22.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3258962944 bytes 2024-09-16T23:13:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3283795968 bytes 2024-09-16T23:13:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3351475200 bytes 2024-09-16T23:13:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3377933312 bytes 2024-09-16T23:13:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3391593472 bytes 2024-09-16T23:13:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3413026816 bytes 2024-09-16T23:13:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3437360128 bytes 2024-09-16T23:13:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3539267584 bytes 2024-09-16T23:13:23.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3576963072 bytes 2024-09-16T23:13:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3646750720 bytes 2024-09-16T23:13:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3658505216 bytes 2024-09-16T23:13:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3696290816 bytes 2024-09-16T23:13:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3753977856 bytes 2024-09-16T23:13:23.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3790997504 bytes 2024-09-16T23:13:23.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3830311936 bytes 2024-09-16T23:13:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3841132544 bytes 2024-09-16T23:13:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3922750464 bytes 2024-09-16T23:13:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3995383808 bytes 2024-09-16T23:13:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 4036270080 bytes 2024-09-16T23:13:24.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4055352320 bytes 2024-09-16T23:13:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4078832640 bytes 2024-09-16T23:13:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4143681536 bytes 2024-09-16T23:13:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4193629184 bytes 2024-09-16T23:13:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4216870912 bytes 2024-09-16T23:13:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4232628224 bytes 2024-09-16T23:13:24.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4275534848 bytes 2024-09-16T23:13:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4364780544 bytes 2024-09-16T23:13:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4395525120 bytes 2024-09-16T23:13:25.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4457487360 bytes 2024-09-16T23:13:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4491123712 bytes 2024-09-16T23:13:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4522949632 bytes 2024-09-16T23:13:25.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4554534912 bytes 2024-09-16T23:13:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4591608832 bytes 2024-09-16T23:13:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4659856384 bytes 2024-09-16T23:13:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4729704448 bytes 2024-09-16T23:13:25.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4766147584 bytes 2024-09-16T23:13:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4766750720 bytes 2024-09-16T23:13:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4794427392 bytes 2024-09-16T23:13:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4887287808 bytes 2024-09-16T23:13:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4887918592 bytes 2024-09-16T23:13:26.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4968672256 bytes 2024-09-16T23:13:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 5004169216 bytes 2024-09-16T23:13:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 5067081728 bytes 2024-09-16T23:13:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 5100759040 bytes 2024-09-16T23:13:26.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 5122042880 bytes 2024-09-16T23:13:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 5159278592 bytes 2024-09-16T23:13:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 5238177792 bytes 2024-09-16T23:13:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 5309285376 bytes 2024-09-16T23:13:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 5330297856 bytes 2024-09-16T23:13:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 5407031296 bytes 2024-09-16T23:13:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5436098560 bytes 2024-09-16T23:13:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 5453072384 bytes 2024-09-16T23:13:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12300 IOs, 5511021568 bytes 2024-09-16T23:13:27.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12400 IOs, 5539571712 bytes 2024-09-16T23:13:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12500 IOs, 5570521088 bytes 2024-09-16T23:13:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12600 IOs, 5581978624 bytes 2024-09-16T23:13:28.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12700 IOs, 5591613440 bytes 2024-09-16T23:13:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12800 IOs, 5636898816 bytes 2024-09-16T23:13:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12900 IOs, 5637504000 bytes 2024-09-16T23:13:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:13:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:13:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:13:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:13:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:13:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:13:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:13:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"} 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:13:00 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:13:28' 2024-09-16T23:13:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:13:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:13:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:13:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"} 2024-09-16T23:13:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:13:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:13:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:13:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:13:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:13:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:13:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:13:29.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:13:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:13:00' 2024-09-16T23:13:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:13:00' 2024-09-16T23:13:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:13:00' 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:13:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:13:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142124314mirror.0smithi09819.3.0-4883-g6e8474a0falseOK42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536233984.0,"last_snapshot_bytes":536082432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528332,"remote_snapshot_timestamp":1726528332,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:13:00cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:13:28' 2024-09-16T23:13:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:13:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:13:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:13:00' 2024-09-16T23:13:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:13:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:13:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:13:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:13:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:13:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap7 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap7 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7 2024-09-16T23:13:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:13:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:13:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:13:59.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:13:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:13:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:13:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:13:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:13:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:13:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:13:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:13:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:13:59.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:13:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:13:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:13:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:13:59.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:13:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:13:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:13:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:13:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:13:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:13:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:13:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:13:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:13:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:13:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:13:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:13:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:13:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:13:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:13:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:13:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:14:01.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 23 2024-09-16T23:14:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:14:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:14:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:14:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:14:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=23 2024-09-16T23:14:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:14:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:01.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 23 = '' 2024-09-16T23:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:14:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:02.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:02.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 23 = '' 2024-09-16T23:14:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 23 = '' 2024-09-16T23:14:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:14:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=23 2024-09-16T23:14:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 23 = 23 2024-09-16T23:14:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:14:04.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:14:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 14 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:14:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:13:11 2024 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:14:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:14:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap6-clean ']' 2024-09-16T23:14:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6-clean 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:06.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:06.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:07.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:07.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:08.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:08.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:08.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap7-clean 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7-clean 2024-09-16T23:14:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:09.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:14:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:14:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:14:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:14:09.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 77903872 bytes 2024-09-16T23:14:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 101422080 bytes 2024-09-16T23:14:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 151480320 bytes 2024-09-16T23:14:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 223960064 bytes 2024-09-16T23:14:10.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 232461312 bytes 2024-09-16T23:14:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 278393856 bytes 2024-09-16T23:14:10.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 309878784 bytes 2024-09-16T23:14:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 354018304 bytes 2024-09-16T23:14:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 420684800 bytes 2024-09-16T23:14:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 468014080 bytes 2024-09-16T23:14:10.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 476711936 bytes 2024-09-16T23:14:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 518199296 bytes 2024-09-16T23:14:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 556369920 bytes 2024-09-16T23:14:11.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 587724800 bytes 2024-09-16T23:14:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 606423040 bytes 2024-09-16T23:14:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 608373760 bytes 2024-09-16T23:14:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 641687552 bytes 2024-09-16T23:14:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 694748160 bytes 2024-09-16T23:14:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 758127616 bytes 2024-09-16T23:14:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 761465856 bytes 2024-09-16T23:14:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 793025536 bytes 2024-09-16T23:14:11.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 832208896 bytes 2024-09-16T23:14:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 907327488 bytes 2024-09-16T23:14:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 941838336 bytes 2024-09-16T23:14:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1019311104 bytes 2024-09-16T23:14:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1090818048 bytes 2024-09-16T23:14:12.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1114468352 bytes 2024-09-16T23:14:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1124370432 bytes 2024-09-16T23:14:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1178208256 bytes 2024-09-16T23:14:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1178807296 bytes 2024-09-16T23:14:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1283247104 bytes 2024-09-16T23:14:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1318447104 bytes 2024-09-16T23:14:13.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1366530048 bytes 2024-09-16T23:14:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1446249472 bytes 2024-09-16T23:14:13.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1479497728 bytes 2024-09-16T23:14:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1517784064 bytes 2024-09-16T23:14:13.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1536838656 bytes 2024-09-16T23:14:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1582748672 bytes 2024-09-16T23:14:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1620604928 bytes 2024-09-16T23:14:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1649690624 bytes 2024-09-16T23:14:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1681623040 bytes 2024-09-16T23:14:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1730191360 bytes 2024-09-16T23:14:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1781376000 bytes 2024-09-16T23:14:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1792873472 bytes 2024-09-16T23:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1837396992 bytes 2024-09-16T23:14:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1928130560 bytes 2024-09-16T23:14:14.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1969553408 bytes 2024-09-16T23:14:14.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2012961792 bytes 2024-09-16T23:14:14.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2102439936 bytes 2024-09-16T23:14:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2203257856 bytes 2024-09-16T23:14:15.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2266078208 bytes 2024-09-16T23:14:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2313316352 bytes 2024-09-16T23:14:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2342669312 bytes 2024-09-16T23:14:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2378050560 bytes 2024-09-16T23:14:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2400379904 bytes 2024-09-16T23:14:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2462317568 bytes 2024-09-16T23:14:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2528370688 bytes 2024-09-16T23:14:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2547707904 bytes 2024-09-16T23:14:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2581064704 bytes 2024-09-16T23:14:16.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2630633472 bytes 2024-09-16T23:14:16.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2726386688 bytes 2024-09-16T23:14:16.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2768368640 bytes 2024-09-16T23:14:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2798121984 bytes 2024-09-16T23:14:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2799611904 bytes 2024-09-16T23:14:16.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2828966912 bytes 2024-09-16T23:14:16.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2882966528 bytes 2024-09-16T23:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2948251648 bytes 2024-09-16T23:14:17.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2984328192 bytes 2024-09-16T23:14:17.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3029630976 bytes 2024-09-16T23:14:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3053750272 bytes 2024-09-16T23:14:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3072128000 bytes 2024-09-16T23:14:17.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:14:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:14:18.431 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T23:14:18.432 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T23:14:18.432 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T23:14:18.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:14:18.435+0000 7f6af6db8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36889) UID: 1000 2024-09-16T23:14:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:14:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:18.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:14:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:14:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:14:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:14:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:14:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536207872.0,"last_snapshot_bytes":536333312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528385,"remote_snapshot_timestamp":1726528385,"replay_state":"idle"} 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:14:00 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:13:58' 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:14:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:14:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536207872.0,"last_snapshot_bytes":536333312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528385,"remote_snapshot_timestamp":1726528385,"replay_state":"idle"} 2024-09-16T23:14:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:14:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:14:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:14:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:14:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:14:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:14:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:14:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:14:00' 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:14:00' 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:14:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:14:00' 2024-09-16T23:14:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:14:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:14:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='UNKNOWNUNKNOWNOK14212mirror.0smithi09819.3.0-4883-g6e8474a0falseUNKNOWN42244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":536207872.0,"last_snapshot_bytes":536333312,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528385,"remote_snapshot_timestamp":1726528385,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:14:00cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:13:58' 2024-09-16T23:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:14:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:14:00' 2024-09-16T23:14:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:14:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:14:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:14:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:14:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:14:18.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap8 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap8 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8 2024-09-16T23:14:18.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:14:19.327 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:14:19.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:14:19.327 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T23:14:19.328 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:14:19.331+0000 7f815c2e6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 38224) UID: 1000 2024-09-16T23:14:22.328 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:14:22.329 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:14:22.329 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:14:22.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:14:22.329 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:14:22.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:14:22.332 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-16T23:14:25.433 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T23:14:25.433 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:14:25.433 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T23:14:25.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T23:14:25.433 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T23:14:25.436 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T23:14:25.436 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:14:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:14:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:14:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:14:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:14:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:14:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:14:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:14:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:14:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:14:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:14:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:14:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:14:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:14:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:14:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:14:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:14:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:14:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:14:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:14:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:14:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:14:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:14:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:14:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:14:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:14:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:14:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:14:49.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:14:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:14:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:14:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:14:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:14:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:14:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:14:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:14:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:14:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:14:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:14:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:14:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:14:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:14:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 26 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:14:50.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:14:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:14:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=26 2024-09-16T23:14:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:51.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:14:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 26 = '' 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:14:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:51.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 26 = '' 2024-09-16T23:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:14:52.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:14:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:14:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:14:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:14:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:14:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 26 = '' 2024-09-16T23:14:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:14:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:14:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:14:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:14:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=26 2024-09-16T23:14:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 26 = 26 2024-09-16T23:14:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:14:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:14:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-16T23:14:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:14:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 16 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:13:11 2024 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:14:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:14:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:14:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:14:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:14:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap7-clean ']' 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7-clean 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:56.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T23:14:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8 2024-09-16T23:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:14:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:14:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap8-clean 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8-clean 2024-09-16T23:14:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:14:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:14:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=9 2024-09-16T23:14:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:14:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:14:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 20108288 bytes 2024-09-16T23:14:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 48911360 bytes 2024-09-16T23:14:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 90620928 bytes 2024-09-16T23:14:59.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 119316480 bytes 2024-09-16T23:14:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 183442432 bytes 2024-09-16T23:14:59.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 184062976 bytes 2024-09-16T23:14:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 203432960 bytes 2024-09-16T23:14:59.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 251109376 bytes 2024-09-16T23:15:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 305600512 bytes 2024-09-16T23:15:00.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 354949120 bytes 2024-09-16T23:15:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 370882560 bytes 2024-09-16T23:15:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 418096128 bytes 2024-09-16T23:15:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 474706944 bytes 2024-09-16T23:15:00.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 550808576 bytes 2024-09-16T23:15:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 598055936 bytes 2024-09-16T23:15:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 604738560 bytes 2024-09-16T23:15:00.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 683421696 bytes 2024-09-16T23:15:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 705302528 bytes 2024-09-16T23:15:01.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 719642624 bytes 2024-09-16T23:15:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 746961920 bytes 2024-09-16T23:15:01.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 765603840 bytes 2024-09-16T23:15:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 834742272 bytes 2024-09-16T23:15:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 892662784 bytes 2024-09-16T23:15:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 947348480 bytes 2024-09-16T23:15:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 948730880 bytes 2024-09-16T23:15:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 978842624 bytes 2024-09-16T23:15:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1001686016 bytes 2024-09-16T23:15:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1047000064 bytes 2024-09-16T23:15:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1093381120 bytes 2024-09-16T23:15:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1134091264 bytes 2024-09-16T23:15:02.363 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:15:02.365+0000 7fbcef505640 -1 reset not still connected to 0x56517cf28680 2024-09-16T23:15:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1186509824 bytes 2024-09-16T23:15:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1214960640 bytes 2024-09-16T23:15:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1223174144 bytes 2024-09-16T23:15:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1312986112 bytes 2024-09-16T23:15:02.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1401742336 bytes 2024-09-16T23:15:02.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1439411200 bytes 2024-09-16T23:15:03.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1467079680 bytes 2024-09-16T23:15:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1490366464 bytes 2024-09-16T23:15:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1493215232 bytes 2024-09-16T23:15:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1498257408 bytes 2024-09-16T23:15:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1530648576 bytes 2024-09-16T23:15:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1563545600 bytes 2024-09-16T23:15:03.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1576238080 bytes 2024-09-16T23:15:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1603087360 bytes 2024-09-16T23:15:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1657921536 bytes 2024-09-16T23:15:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1693450240 bytes 2024-09-16T23:15:04.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1754138624 bytes 2024-09-16T23:15:04.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1789195264 bytes 2024-09-16T23:15:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1829541888 bytes 2024-09-16T23:15:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1851220992 bytes 2024-09-16T23:15:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1896742912 bytes 2024-09-16T23:15:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 1934462976 bytes 2024-09-16T23:15:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 1959301120 bytes 2024-09-16T23:15:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2007003136 bytes 2024-09-16T23:15:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2032189440 bytes 2024-09-16T23:15:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2040698880 bytes 2024-09-16T23:15:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2073068544 bytes 2024-09-16T23:15:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2123682816 bytes 2024-09-16T23:15:05.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2187192320 bytes 2024-09-16T23:15:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2222944256 bytes 2024-09-16T23:15:05.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2252971008 bytes 2024-09-16T23:15:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2321400832 bytes 2024-09-16T23:15:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2322982912 bytes 2024-09-16T23:15:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2407496704 bytes 2024-09-16T23:15:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2408817664 bytes 2024-09-16T23:15:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2455384064 bytes 2024-09-16T23:15:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2535900160 bytes 2024-09-16T23:15:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2553822208 bytes 2024-09-16T23:15:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2566022144 bytes 2024-09-16T23:15:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2624413696 bytes 2024-09-16T23:15:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2689602560 bytes 2024-09-16T23:15:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2747699200 bytes 2024-09-16T23:15:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2777619456 bytes 2024-09-16T23:15:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2818079744 bytes 2024-09-16T23:15:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2822834176 bytes 2024-09-16T23:15:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2886556672 bytes 2024-09-16T23:15:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 2949666816 bytes 2024-09-16T23:15:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 2978391040 bytes 2024-09-16T23:15:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3070168064 bytes 2024-09-16T23:15:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3086991360 bytes 2024-09-16T23:15:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3094941696 bytes 2024-09-16T23:15:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3157095424 bytes 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:15:07.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:15:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:15:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:15:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:15:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"} 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:15:00 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:14:41' 2024-09-16T23:15:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:15:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:15:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:15:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"} 2024-09-16T23:15:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:15:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:15:08.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:15:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:15:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:15:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:15:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:15:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:15:00' 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:15:00' 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:15:00' 2024-09-16T23:15:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:15:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:15:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:15:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOK45604563mirror.0smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:15:00cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:14:41' 2024-09-16T23:15:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:15:09.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:15:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:15:00' 2024-09-16T23:15:09.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:15:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:15:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:15:09.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:15:09.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:15:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap9 2024-09-16T23:15:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap9 2024-09-16T23:15:09.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9 2024-09-16T23:15:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:15:29.333 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.286934789663402) > weight (0.25) 2024-09-16T23:15:29.334 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6037191666219811) > weight (0.25) 2024-09-16T23:15:29.334 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T23:15:29.334 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T23:15:29.335 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T23:15:29.335 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:15:29.338+0000 7f0202c26640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 36987) UID: 1000 2024-09-16T23:15:35.335 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T23:15:35.336 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:15:35.336 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T23:15:35.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:15:35.336 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:15:35.340 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:15:35.340 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:15:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:15:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:15:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:15:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:15:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:15:37.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:15:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:15:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:15:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:15:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:15:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:15:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:15:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:15:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:15:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:15:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:15:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:15:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:15:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:15:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:15:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:15:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:15:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:15:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:15:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:15:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:15:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:15:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:15:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:15:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:15:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:15:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:15:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 29 2024-09-16T23:15:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:15:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:15:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:15:39.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:15:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:15:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=29 2024-09-16T23:15:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:15:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:15:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:15:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:15:40.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:15:40.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:15:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:15:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:15:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:15:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:15:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 29 = '' 2024-09-16T23:15:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:15:40.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:15:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:15:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:15:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:15:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 29 = '' 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:15:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:15:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:15:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:15:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:15:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 29 = '' 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:15:41.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:15:42.006 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:15:42.009+0000 7fbcef505640 -1 reset not still connected to 0x56517b0c9ee0 2024-09-16T23:15:42.008 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:15:42.011+0000 7fbcef505640 -1 reset not still connected to 0x56517ca6cdd0 2024-09-16T23:15:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:15:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:15:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:15:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:15:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=29 2024-09-16T23:15:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 29 = 29 2024-09-16T23:15:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-16T23:15:43.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:15:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:15:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 18 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:15:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:14:55 2024 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:15:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap8-clean ']' 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8-clean 2024-09-16T23:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:15:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:15:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:45.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:15:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9 2024-09-16T23:15:46.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:15:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:15:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:46.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:15:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:15:46.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:47.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:15:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:15:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap9-clean 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9-clean 2024-09-16T23:15:47.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror test 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:47.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:15:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:15:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:15:47.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-16T23:15:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 74918912 bytes 2024-09-16T23:15:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 103327744 bytes 2024-09-16T23:15:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 155120640 bytes 2024-09-16T23:15:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 155732992 bytes 2024-09-16T23:15:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 231106560 bytes 2024-09-16T23:15:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 297029632 bytes 2024-09-16T23:15:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 378672128 bytes 2024-09-16T23:15:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 388955136 bytes 2024-09-16T23:15:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 409666560 bytes 2024-09-16T23:15:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 457221120 bytes 2024-09-16T23:15:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 497946624 bytes 2024-09-16T23:15:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 498544640 bytes 2024-09-16T23:15:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 506511360 bytes 2024-09-16T23:15:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 554635264 bytes 2024-09-16T23:15:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 641214464 bytes 2024-09-16T23:15:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 690011136 bytes 2024-09-16T23:15:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 704842752 bytes 2024-09-16T23:15:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 744086528 bytes 2024-09-16T23:15:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 811236352 bytes 2024-09-16T23:15:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 874352640 bytes 2024-09-16T23:15:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 878990336 bytes 2024-09-16T23:15:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 929002496 bytes 2024-09-16T23:15:50.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 946664448 bytes 2024-09-16T23:15:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1055351808 bytes 2024-09-16T23:15:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1101400064 bytes 2024-09-16T23:15:51.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1113129984 bytes 2024-09-16T23:15:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1171831808 bytes 2024-09-16T23:15:51.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1213504512 bytes 2024-09-16T23:15:51.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1236365312 bytes 2024-09-16T23:15:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1280191488 bytes 2024-09-16T23:15:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1307780096 bytes 2024-09-16T23:15:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1387234304 bytes 2024-09-16T23:15:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1438715904 bytes 2024-09-16T23:15:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1440176128 bytes 2024-09-16T23:15:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1495573504 bytes 2024-09-16T23:15:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1525117952 bytes 2024-09-16T23:15:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1641965568 bytes 2024-09-16T23:15:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1693669376 bytes 2024-09-16T23:15:52.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1734517760 bytes 2024-09-16T23:15:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1779004416 bytes 2024-09-16T23:15:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1779635200 bytes 2024-09-16T23:15:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1797331968 bytes 2024-09-16T23:15:53.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1861836800 bytes 2024-09-16T23:15:53.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1893087232 bytes 2024-09-16T23:15:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1915862016 bytes 2024-09-16T23:15:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2026518528 bytes 2024-09-16T23:15:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2073014272 bytes 2024-09-16T23:15:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2138787840 bytes 2024-09-16T23:15:53.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2166853632 bytes 2024-09-16T23:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2217493504 bytes 2024-09-16T23:15:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:15:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:15:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:15:53.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:15:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:15:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-16T23:15:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local description_pattern= 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local service_pattern= 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:15:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-16T23:15:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='test: 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: global_id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+replaying 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: replaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"} 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: service: mirror.1 on smithi098 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:15:30 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: peer_sites: 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: name: cluster2 2024-09-16T23:15:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: state: up+stopped 2024-09-16T23:15:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: description: local image is primary 2024-09-16T23:15:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: last_update: 2024-09-16 23:15:41' 2024-09-16T23:15:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T23:15:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: state: up+replaying 2024-09-16T23:15:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^ description: .*' 2024-09-16T23:15:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: description: replaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"} 2024-09-16T23:15:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:15:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:15:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep '^up+' 2024-09-16T23:15:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:15:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T23:15:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: service: mirror.1 on smithi098 2024-09-16T23:15:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update 2024-09-16T23:15:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:15:30' 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-16 23:15:30' 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state_pattern=up+replaying 2024-09-16T23:15:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local 'prev_last_update=2024-09-16 23:15:30' 2024-09-16T23:15:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status 2024-09-16T23:15:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ CEPH_ARGS= 2024-09-16T23:15:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T23:15:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status='OKOKOK142244311mirror.3smithi09819.3.0-4883-g6e8474a0falseOK42944297mirror.1smithi09819.3.0-4883-g6e8474a0trueOK44354438mirror.2smithi09819.3.0-4883-g6e8474a0falseOK45604563mirror.0smithi09819.3.0-4883-g6e8474a0falseOKtest3d821a06-9725-453f-825f-47ca45e6769aup+replayingreplaying, {"bytes_per_second":17804800.0,"bytes_per_snapshot":534470656.0,"last_snapshot_bytes":534144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726528489,"remote_snapshot_timestamp":1726528489,"replay_state":"idle"}42944297mirror.1smithi0982024-09-16 23:15:30cluster2719c1107-0abe-4e5e-9d82-f041ee6c7a04up+stoppedlocal image is primary2024-09-16 23:15:41' 2024-09-16T23:15:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_update state 2024-09-16T23:15:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-16T23:15:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ last_update='2024-09-16 23:15:30' 2024-09-16T23:15:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-16T23:15:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=up+replaying 2024-09-16T23:15:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo up+replaying 2024-09-16T23:15:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep up+replaying 2024-09-16T23:15:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:up+replaying 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap10 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap10 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:15:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:15:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:15:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10 2024-09-16T23:15:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:16:08.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.45026226038881845) > weight (0.25) 2024-09-16T23:16:08.437 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.42133653173709185) > weight (0.25) 2024-09-16T23:16:08.437 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T23:16:08.438 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T23:16:08.438 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-16T23:16:08.439 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:16:08.442+0000 7fa14b34a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40375) UID: 1000 2024-09-16T23:16:10.439 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T23:16:10.439 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:16:10.440 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T23:16:10.440 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T23:16:10.440 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T23:16:10.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T23:16:10.443 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-16T23:16:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:16:23.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:16:24.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:16:24.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:16:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:16:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:16:24.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:16:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:16:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:16:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:25.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:16:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:16:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:16:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:16:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:16:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:16:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:16:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:16:25.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:16:25.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:16:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:16:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:16:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:16:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:16:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:16:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:16:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:16:25.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:16:25.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:16:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 32 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:16:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:16:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=32 2024-09-16T23:16:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:16:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:16:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:16:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:16:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:16:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 32 = '' 2024-09-16T23:16:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:16:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:27.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:16:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:16:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:16:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:16:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 32 = '' 2024-09-16T23:16:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:27.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:16:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:16:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:16:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:16:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 32 = '' 2024-09-16T23:16:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:28.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:16:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=32 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 32 = 32 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:16:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:16:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 20 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:16:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:14:55 2024 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:16:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:16:31.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snap9-clean ']' 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9-clean 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:31.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:31.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:33.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:33.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ clean_snap_name=snap10-clean 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10-clean 2024-09-16T23:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:16:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:16:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:16:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:16:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-16T23:16:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:16:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:16:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:16:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:16:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:16:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:16:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:16:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:16:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:16:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:16:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:16:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:16:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:16:35.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:16:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:35.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:16:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-16T23:16:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-16T23:16:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:16:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:16:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:16:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:16:36.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:16:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:16:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:16:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:16:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-16T23:16:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:16:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:16:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:16:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:16:36.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:16:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:16:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:16:36.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:16:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-test.status 2024-09-16T23:16:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-test.status 2024-09-16T23:16:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:16:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-test.status 2024-09-16T23:16:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-test.status 2024-09-16T23:16:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:16:36.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-16T23:16:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:36.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-16T23:16:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 34 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.status 2024-09-16T23:16:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-16T23:16:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:16:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=34 2024-09-16T23:16:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:16:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:16:38.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:16:38.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:16:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:16:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:16:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 34 = '' 2024-09-16T23:16:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:16:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:16:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.status 2024-09-16T23:16:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-16T23:16:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:16:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=34 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 34 = 34 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:16:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10-clean 2024-09-16T23:16:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:16:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:16:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:16:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-16T23:16:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'test': 2024-09-16T23:16:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:16:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:16:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 21 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c925039ff 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c925039ff 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:16:32 2024 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:08:02 2024 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:16:39.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3d821a06-9725-453f-825f-47ca45e6769a 2024-09-16T23:16:39.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:16:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 10 2024-09-16T23:16:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap1 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap1 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:39.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:40.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap1-clean 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap1-clean 2024-09-16T23:16:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:41.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:42.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap2 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap2 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:43.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:44.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap2-clean 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap2-clean 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:45.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap3 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap3 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3 2024-09-16T23:16:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap3-clean 2024-09-16T23:16:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-16T23:16:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap3-clean 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:47.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:48.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap4 2024-09-16T23:16:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap4 2024-09-16T23:16:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:49.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap4-clean 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap4-clean 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap5 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap5 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:52.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:53.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:53.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:53.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap5-clean 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap5-clean 2024-09-16T23:16:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:54.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:55.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap6 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap6 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:55.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:57.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap6-clean 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap6-clean 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:58.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:58.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:16:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:16:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap7 2024-09-16T23:16:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap7 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:16:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:16:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:16:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:00.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap7-clean 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap7-clean 2024-09-16T23:17:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:01.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:01.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap8 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap8 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:03.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap8-clean 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap8-clean 2024-09-16T23:17:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:04.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:04.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap9 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap9 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:06.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:06.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap9-clean 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap9-clean 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap10 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap10 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:09.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:09.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap10-clean 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap10-clean 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:10.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export 2024-09-16T23:17:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-16T23:17:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-test.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-test.export 2024-09-16T23:17:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 10 2024-09-16T23:17:11.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap1 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap1 2024-09-16T23:17:11.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-16T23:17:11.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap1-clean 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap1-clean 2024-09-16T23:17:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-16T23:17:12.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap2 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap2 2024-09-16T23:17:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-16T23:17:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap2-clean 2024-09-16T23:17:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-16T23:17:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap2-clean 2024-09-16T23:17:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-16T23:17:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap3 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap3 2024-09-16T23:17:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-16T23:17:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap3-clean 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap3-clean 2024-09-16T23:17:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-16T23:17:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap4 2024-09-16T23:17:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-16T23:17:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap4 2024-09-16T23:17:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-16T23:17:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap4-clean 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap4-clean 2024-09-16T23:17:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-16T23:17:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:19.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap5 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap5 2024-09-16T23:17:19.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-16T23:17:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap5-clean 2024-09-16T23:17:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-16T23:17:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap5-clean 2024-09-16T23:17:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-16T23:17:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap6 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap6 2024-09-16T23:17:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-16T23:17:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap6-clean 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap6-clean 2024-09-16T23:17:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-16T23:17:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap7 2024-09-16T23:17:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-16T23:17:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap7 2024-09-16T23:17:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-16T23:17:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap7-clean 2024-09-16T23:17:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-16T23:17:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap7-clean 2024-09-16T23:17:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-16T23:17:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap8 2024-09-16T23:17:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-16T23:17:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap8 2024-09-16T23:17:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-16T23:17:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap8-clean 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap8-clean 2024-09-16T23:17:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-16T23:17:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap9 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap9 2024-09-16T23:17:27.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-16T23:17:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap9-clean 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap9-clean 2024-09-16T23:17:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-16T23:17:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:29.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap10 2024-09-16T23:17:29.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-16T23:17:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap10 2024-09-16T23:17:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-16T23:17:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap10-clean 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap10-clean 2024-09-16T23:17:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-16T23:17:31.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror test 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:17:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:17:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror test 2024-09-16T23:17:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-16T23:17:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-16T23:17:31.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:17:31.198+0000 7fabad865640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a6a9994820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:17:31.196 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:17:31.198+0000 7fabb3871640 -1 rbd::mirror::ImageReplayer: 0x55a6a9946000 [3/3d821a06-9725-453f-825f-47ca45e6769a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:17:31.340 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:17:31.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:17:31.341 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T23:17:31.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:17:31.344+0000 7fb73351f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 42122) UID: 1000 2024-09-16T23:17:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:17:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-16T23:17:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:17:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror test 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=test 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T23:17:32.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:17:32.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T23:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:17:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=test 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:17:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror test 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=test 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-16T23:17:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:17:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: create many images' 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:17:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:17:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:17:32 TEST: create many images 2024-09-16T23:17:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:17:32 TEST: create many images 2024-09-16T23:17:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ snap_name=snap 2024-09-16T23:17:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 50 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_1 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:17:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:17:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:17:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2024-09-16T23:17:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:17:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T23:17:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:17:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T23:17:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_1': 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 126bee81a140 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.126bee81a140 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:32 2024 2024-09-16T23:17:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:32 2024 2024-09-16T23:17:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:32 2024 2024-09-16T23:17:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:17:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:17:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2fda3e9f-ae26-4e8f-b0fd-5a14428de02e 2024-09-16T23:17:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:17:33.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:17:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:17:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:17:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:17:33.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 65078272 bytes 2024-09-16T23:17:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 119992320 bytes 2024-09-16T23:17:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 163387392 bytes 2024-09-16T23:17:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 188412928 bytes 2024-09-16T23:17:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 232933376 bytes 2024-09-16T23:17:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 285849600 bytes 2024-09-16T23:17:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 332397568 bytes 2024-09-16T23:17:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 350501888 bytes 2024-09-16T23:17:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 411013120 bytes 2024-09-16T23:17:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 464680960 bytes 2024-09-16T23:17:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 480356352 bytes 2024-09-16T23:17:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 534602752 bytes 2024-09-16T23:17:34.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 559783936 bytes 2024-09-16T23:17:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 592645120 bytes 2024-09-16T23:17:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 627346432 bytes 2024-09-16T23:17:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 671558656 bytes 2024-09-16T23:17:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 726777856 bytes 2024-09-16T23:17:35.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 775404544 bytes 2024-09-16T23:17:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 839413760 bytes 2024-09-16T23:17:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 877361152 bytes 2024-09-16T23:17:35.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 892065792 bytes 2024-09-16T23:17:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 952796160 bytes 2024-09-16T23:17:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1006722048 bytes 2024-09-16T23:17:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1031804928 bytes 2024-09-16T23:17:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1100344320 bytes 2024-09-16T23:17:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1147624448 bytes 2024-09-16T23:17:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1280200704 bytes 2024-09-16T23:17:36.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1333332992 bytes 2024-09-16T23:17:36.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1372852224 bytes 2024-09-16T23:17:36.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1394331648 bytes 2024-09-16T23:17:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1453509632 bytes 2024-09-16T23:17:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1472169984 bytes 2024-09-16T23:17:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1538029568 bytes 2024-09-16T23:17:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1577593856 bytes 2024-09-16T23:17:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1614791680 bytes 2024-09-16T23:17:36.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1639384064 bytes 2024-09-16T23:17:37.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1662395392 bytes 2024-09-16T23:17:37.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1662995456 bytes 2024-09-16T23:17:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1713724416 bytes 2024-09-16T23:17:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1808441344 bytes 2024-09-16T23:17:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1817935872 bytes 2024-09-16T23:17:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1828324352 bytes 2024-09-16T23:17:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1918968832 bytes 2024-09-16T23:17:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1928365056 bytes 2024-09-16T23:17:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1968881664 bytes 2024-09-16T23:17:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2026885120 bytes 2024-09-16T23:17:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2070909952 bytes 2024-09-16T23:17:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2098096128 bytes 2024-09-16T23:17:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2169619456 bytes 2024-09-16T23:17:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2189972480 bytes 2024-09-16T23:17:38.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2208064512 bytes 2024-09-16T23:17:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2260235264 bytes 2024-09-16T23:17:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2260856832 bytes 2024-09-16T23:17:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2341511168 bytes 2024-09-16T23:17:38.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2375260160 bytes 2024-09-16T23:17:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2456701952 bytes 2024-09-16T23:17:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2553851904 bytes 2024-09-16T23:17:39.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2591273984 bytes 2024-09-16T23:17:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2619602944 bytes 2024-09-16T23:17:39.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2659869696 bytes 2024-09-16T23:17:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2683668480 bytes 2024-09-16T23:17:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2704135168 bytes 2024-09-16T23:17:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2712070144 bytes 2024-09-16T23:17:39.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2714288128 bytes 2024-09-16T23:17:39.892 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:17:39.893+0000 7f440f149640 -1 reset not still connected to 0x55f26bbf6d00 2024-09-16T23:17:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2746764288 bytes 2024-09-16T23:17:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2784565248 bytes 2024-09-16T23:17:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_2 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:17:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:17:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:17:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T23:17:40.342 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:17:40.343 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:17:40.343 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:17:40.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:17:40.343 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:17:40.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:17:40.346 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-16T23:17:40.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:17:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_2': 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1276904f2d16 2024-09-16T23:17:41.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1276904f2d16 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:40 2024 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:40 2024 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:40 2024 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 72e6208d-3742-4bfa-8918-249d11759c7e 2024-09-16T23:17:41.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:17:41.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:17:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:17:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:17:41.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:17:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 81162240 bytes 2024-09-16T23:17:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 81740800 bytes 2024-09-16T23:17:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 120845312 bytes 2024-09-16T23:17:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 121470976 bytes 2024-09-16T23:17:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 204912640 bytes 2024-09-16T23:17:41.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 208570368 bytes 2024-09-16T23:17:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 278798336 bytes 2024-09-16T23:17:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 280087552 bytes 2024-09-16T23:17:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 312637440 bytes 2024-09-16T23:17:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 372684800 bytes 2024-09-16T23:17:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 402962432 bytes 2024-09-16T23:17:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 423777280 bytes 2024-09-16T23:17:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 439738368 bytes 2024-09-16T23:17:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 467274752 bytes 2024-09-16T23:17:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 531034112 bytes 2024-09-16T23:17:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 587633664 bytes 2024-09-16T23:17:42.649 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:17:42.651+0000 7f440f149640 -1 reset not still connected to 0x55f26b103380 2024-09-16T23:17:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 594257920 bytes 2024-09-16T23:17:42.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 698501120 bytes 2024-09-16T23:17:42.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 738117632 bytes 2024-09-16T23:17:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 799661056 bytes 2024-09-16T23:17:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 851792896 bytes 2024-09-16T23:17:43.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 858741760 bytes 2024-09-16T23:17:43.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 926084096 bytes 2024-09-16T23:17:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 936988672 bytes 2024-09-16T23:17:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1033882624 bytes 2024-09-16T23:17:43.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1097392128 bytes 2024-09-16T23:17:43.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1118022656 bytes 2024-09-16T23:17:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1223391232 bytes 2024-09-16T23:17:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1260082176 bytes 2024-09-16T23:17:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1277801472 bytes 2024-09-16T23:17:44.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1293537280 bytes 2024-09-16T23:17:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1323446272 bytes 2024-09-16T23:17:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1381339136 bytes 2024-09-16T23:17:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1440543744 bytes 2024-09-16T23:17:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1482806272 bytes 2024-09-16T23:17:44.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1500771328 bytes 2024-09-16T23:17:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1581762560 bytes 2024-09-16T23:17:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1639072768 bytes 2024-09-16T23:17:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1686549504 bytes 2024-09-16T23:17:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1720380416 bytes 2024-09-16T23:17:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1733368832 bytes 2024-09-16T23:17:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1733994496 bytes 2024-09-16T23:17:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1788559360 bytes 2024-09-16T23:17:45.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1819915264 bytes 2024-09-16T23:17:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1851404288 bytes 2024-09-16T23:17:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1891843072 bytes 2024-09-16T23:17:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1953275904 bytes 2024-09-16T23:17:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1961407488 bytes 2024-09-16T23:17:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2022828032 bytes 2024-09-16T23:17:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2034907136 bytes 2024-09-16T23:17:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2089033728 bytes 2024-09-16T23:17:46.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2119142400 bytes 2024-09-16T23:17:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2161301504 bytes 2024-09-16T23:17:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2170880000 bytes 2024-09-16T23:17:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2215795712 bytes 2024-09-16T23:17:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2252004352 bytes 2024-09-16T23:17:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2299855872 bytes 2024-09-16T23:17:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2359230464 bytes 2024-09-16T23:17:47.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2409900032 bytes 2024-09-16T23:17:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2449722368 bytes 2024-09-16T23:17:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2506375168 bytes 2024-09-16T23:17:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2534153216 bytes 2024-09-16T23:17:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2597628928 bytes 2024-09-16T23:17:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2669126656 bytes 2024-09-16T23:17:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2681676800 bytes 2024-09-16T23:17:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2696938496 bytes 2024-09-16T23:17:48.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2738734080 bytes 2024-09-16T23:17:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2745915392 bytes 2024-09-16T23:17:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2782511104 bytes 2024-09-16T23:17:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2821186560 bytes 2024-09-16T23:17:48.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4590465978848104) > weight (0.25) 2024-09-16T23:17:48.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8832105265159058) > weight (0.25) 2024-09-16T23:17:48.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8240659663709627) > weight (0.25) 2024-09-16T23:17:48.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.29633837797869755) > weight (0.25) 2024-09-16T23:17:48.444 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2024-09-16T23:17:48.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2944351232 bytes 2024-09-16T23:17:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3033682944 bytes 2024-09-16T23:17:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3079394304 bytes 2024-09-16T23:17:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3199091712 bytes 2024-09-16T23:17:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3259154432 bytes 2024-09-16T23:17:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3308488704 bytes 2024-09-16T23:17:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3353676800 bytes 2024-09-16T23:17:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3381090304 bytes 2024-09-16T23:17:49.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3451897856 bytes 2024-09-16T23:17:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3480854528 bytes 2024-09-16T23:17:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3559721984 bytes 2024-09-16T23:17:49.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3583494144 bytes 2024-09-16T23:17:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3584123904 bytes 2024-09-16T23:17:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3608025088 bytes 2024-09-16T23:17:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3674018816 bytes 2024-09-16T23:17:50.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3694315520 bytes 2024-09-16T23:17:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3698700288 bytes 2024-09-16T23:17:50.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3730171904 bytes 2024-09-16T23:17:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3807156224 bytes 2024-09-16T23:17:50.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3846531072 bytes 2024-09-16T23:17:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3920022528 bytes 2024-09-16T23:17:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3939111936 bytes 2024-09-16T23:17:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4004752384 bytes 2024-09-16T23:17:51.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4043654144 bytes 2024-09-16T23:17:51.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4055205888 bytes 2024-09-16T23:17:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4098801664 bytes 2024-09-16T23:17:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4139409408 bytes 2024-09-16T23:17:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4177350656 bytes 2024-09-16T23:17:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4219757568 bytes 2024-09-16T23:17:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4253484032 bytes 2024-09-16T23:17:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4254814208 bytes 2024-09-16T23:17:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4309324800 bytes 2024-09-16T23:17:51.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4316546048 bytes 2024-09-16T23:17:52.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4351521792 bytes 2024-09-16T23:17:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4401996800 bytes 2024-09-16T23:17:52.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4418876416 bytes 2024-09-16T23:17:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4465348608 bytes 2024-09-16T23:17:52.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4534325248 bytes 2024-09-16T23:17:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4534946816 bytes 2024-09-16T23:17:52.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4560801792 bytes 2024-09-16T23:17:52.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4598394880 bytes 2024-09-16T23:17:52.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4632057856 bytes 2024-09-16T23:17:52.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4670940160 bytes 2024-09-16T23:17:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4715954176 bytes 2024-09-16T23:17:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 4753999872 bytes 2024-09-16T23:17:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 4781243392 bytes 2024-09-16T23:17:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 4844760064 bytes 2024-09-16T23:17:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 4943820800 bytes 2024-09-16T23:17:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 4970895360 bytes 2024-09-16T23:17:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 4979227648 bytes 2024-09-16T23:17:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5069146112 bytes 2024-09-16T23:17:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12200 IOs, 5101741056 bytes 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_3 2024-09-16T23:17:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-16T23:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:17:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2024-09-16T23:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:17:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-16T23:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:17:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T23:17:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:17:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_3': 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 128bd3197af5 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.128bd3197af5 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:17:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:54 2024 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:54 2024 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:54 2024 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 7dc49f0a-1a35-4bab-aea4-16de6b2f77bc 2024-09-16T23:17:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:17:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:17:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T23:17:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:17:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:17:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:17:55.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:17:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:17:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:17:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:17:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 29233152 bytes 2024-09-16T23:17:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 57962496 bytes 2024-09-16T23:17:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 112608256 bytes 2024-09-16T23:17:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 126027776 bytes 2024-09-16T23:17:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 191681536 bytes 2024-09-16T23:17:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 262236160 bytes 2024-09-16T23:17:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 273719296 bytes 2024-09-16T23:17:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 307141632 bytes 2024-09-16T23:17:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 312382464 bytes 2024-09-16T23:17:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 319192064 bytes 2024-09-16T23:17:56.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 370525184 bytes 2024-09-16T23:17:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 371296256 bytes 2024-09-16T23:17:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 410131456 bytes 2024-09-16T23:17:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 430802944 bytes 2024-09-16T23:17:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 487820288 bytes 2024-09-16T23:17:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 555176960 bytes 2024-09-16T23:17:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 616879104 bytes 2024-09-16T23:17:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 651779072 bytes 2024-09-16T23:17:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 681062400 bytes 2024-09-16T23:17:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 690014208 bytes 2024-09-16T23:17:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 725016576 bytes 2024-09-16T23:17:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 787321856 bytes 2024-09-16T23:17:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 798040064 bytes 2024-09-16T23:17:57.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 839096320 bytes 2024-09-16T23:17:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 850292736 bytes 2024-09-16T23:17:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 885419008 bytes 2024-09-16T23:17:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 919294976 bytes 2024-09-16T23:17:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 949394432 bytes 2024-09-16T23:17:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 950005760 bytes 2024-09-16T23:17:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1011776512 bytes 2024-09-16T23:17:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1108913152 bytes 2024-09-16T23:17:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1134692352 bytes 2024-09-16T23:17:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1168632832 bytes 2024-09-16T23:17:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1232976896 bytes 2024-09-16T23:17:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1325164544 bytes 2024-09-16T23:17:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1342636032 bytes 2024-09-16T23:17:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1404322816 bytes 2024-09-16T23:17:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1426131968 bytes 2024-09-16T23:17:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1514885120 bytes 2024-09-16T23:17:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1522112512 bytes 2024-09-16T23:17:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1617996800 bytes 2024-09-16T23:17:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1621606400 bytes 2024-09-16T23:17:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1675073536 bytes 2024-09-16T23:17:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1724161024 bytes 2024-09-16T23:17:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1767385088 bytes 2024-09-16T23:17:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1846551552 bytes 2024-09-16T23:18:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1951436800 bytes 2024-09-16T23:18:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2040885248 bytes 2024-09-16T23:18:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2092182528 bytes 2024-09-16T23:18:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2125584384 bytes 2024-09-16T23:18:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2206611456 bytes 2024-09-16T23:18:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2207233024 bytes 2024-09-16T23:18:00.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2252240896 bytes 2024-09-16T23:18:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2309540864 bytes 2024-09-16T23:18:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2343692288 bytes 2024-09-16T23:18:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2367413248 bytes 2024-09-16T23:18:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2441288704 bytes 2024-09-16T23:18:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2537710592 bytes 2024-09-16T23:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2582943744 bytes 2024-09-16T23:18:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2664865792 bytes 2024-09-16T23:18:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2697083904 bytes 2024-09-16T23:18:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2697717760 bytes 2024-09-16T23:18:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2742519808 bytes 2024-09-16T23:18:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2743139328 bytes 2024-09-16T23:18:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2812815360 bytes 2024-09-16T23:18:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2813411328 bytes 2024-09-16T23:18:02.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2822096896 bytes 2024-09-16T23:18:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2862196736 bytes 2024-09-16T23:18:02.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2886704128 bytes 2024-09-16T23:18:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2887283712 bytes 2024-09-16T23:18:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2928111616 bytes 2024-09-16T23:18:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2957957120 bytes 2024-09-16T23:18:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2958555136 bytes 2024-09-16T23:18:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3002176512 bytes 2024-09-16T23:18:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3056707584 bytes 2024-09-16T23:18:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3057326080 bytes 2024-09-16T23:18:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3111237632 bytes 2024-09-16T23:18:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3200067584 bytes 2024-09-16T23:18:03.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3246195712 bytes 2024-09-16T23:18:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3252032512 bytes 2024-09-16T23:18:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3339471872 bytes 2024-09-16T23:18:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3376855040 bytes 2024-09-16T23:18:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3422094336 bytes 2024-09-16T23:18:04.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3494545408 bytes 2024-09-16T23:18:04.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3501783040 bytes 2024-09-16T23:18:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3525420032 bytes 2024-09-16T23:18:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3561907200 bytes 2024-09-16T23:18:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3642736640 bytes 2024-09-16T23:18:04.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3691241472 bytes 2024-09-16T23:18:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3712134144 bytes 2024-09-16T23:18:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3753607168 bytes 2024-09-16T23:18:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3799962624 bytes 2024-09-16T23:18:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3907232768 bytes 2024-09-16T23:18:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3912069120 bytes 2024-09-16T23:18:05.355 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3942937600 bytes 2024-09-16T23:18:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3951269888 bytes 2024-09-16T23:18:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 3973084160 bytes 2024-09-16T23:18:05.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 3983176704 bytes 2024-09-16T23:18:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4050594816 bytes 2024-09-16T23:18:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4139668480 bytes 2024-09-16T23:18:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4160017408 bytes 2024-09-16T23:18:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4192598016 bytes 2024-09-16T23:18:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4223641600 bytes 2024-09-16T23:18:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4232401920 bytes 2024-09-16T23:18:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4279217152 bytes 2024-09-16T23:18:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4298870784 bytes 2024-09-16T23:18:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4375865344 bytes 2024-09-16T23:18:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4388978688 bytes 2024-09-16T23:18:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4396266496 bytes 2024-09-16T23:18:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4412720128 bytes 2024-09-16T23:18:06.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4433580032 bytes 2024-09-16T23:18:07.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4459944960 bytes 2024-09-16T23:18:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4503742464 bytes 2024-09-16T23:18:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4511388672 bytes 2024-09-16T23:18:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 4520074240 bytes 2024-09-16T23:18:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 4580461568 bytes 2024-09-16T23:18:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 4600404992 bytes 2024-09-16T23:18:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 4608880640 bytes 2024-09-16T23:18:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 4647463936 bytes 2024-09-16T23:18:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 4666456064 bytes 2024-09-16T23:18:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 4730727424 bytes 2024-09-16T23:18:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_4 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:18:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T23:18:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:08.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T23:18:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_4': 2024-09-16T23:18:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12978852fb66 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12978852fb66 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:08 2024 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:08 2024 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:08 2024 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8d46cddb-0d98-4f2b-8e9f-54007c183e08 2024-09-16T23:18:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T23:18:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:18:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:18:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:08.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 24078336 bytes 2024-09-16T23:18:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 95722496 bytes 2024-09-16T23:18:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 184813568 bytes 2024-09-16T23:18:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 244745216 bytes 2024-09-16T23:18:08.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 289262592 bytes 2024-09-16T23:18:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 349051904 bytes 2024-09-16T23:18:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 373889024 bytes 2024-09-16T23:18:09.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 440915968 bytes 2024-09-16T23:18:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 463123456 bytes 2024-09-16T23:18:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 502173696 bytes 2024-09-16T23:18:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 550004736 bytes 2024-09-16T23:18:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 561229824 bytes 2024-09-16T23:18:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 623185920 bytes 2024-09-16T23:18:09.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 723570688 bytes 2024-09-16T23:18:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 741525504 bytes 2024-09-16T23:18:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 805728256 bytes 2024-09-16T23:18:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 834480128 bytes 2024-09-16T23:18:10.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 874135552 bytes 2024-09-16T23:18:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 879745024 bytes 2024-09-16T23:18:10.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 910727168 bytes 2024-09-16T23:18:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 927051776 bytes 2024-09-16T23:18:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 961498112 bytes 2024-09-16T23:18:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 989026304 bytes 2024-09-16T23:18:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1001508864 bytes 2024-09-16T23:18:11.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1052279808 bytes 2024-09-16T23:18:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1083402240 bytes 2024-09-16T23:18:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1106210816 bytes 2024-09-16T23:18:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1111288832 bytes 2024-09-16T23:18:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1111900160 bytes 2024-09-16T23:18:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1117462528 bytes 2024-09-16T23:18:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1150262272 bytes 2024-09-16T23:18:11.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1175071744 bytes 2024-09-16T23:18:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1283011584 bytes 2024-09-16T23:18:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1339657216 bytes 2024-09-16T23:18:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1387505664 bytes 2024-09-16T23:18:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1489255424 bytes 2024-09-16T23:18:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1503222784 bytes 2024-09-16T23:18:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1506757632 bytes 2024-09-16T23:18:12.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1589706752 bytes 2024-09-16T23:18:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1639560192 bytes 2024-09-16T23:18:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1684573184 bytes 2024-09-16T23:18:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1716162560 bytes 2024-09-16T23:18:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1780508672 bytes 2024-09-16T23:18:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1827648512 bytes 2024-09-16T23:18:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1829297152 bytes 2024-09-16T23:18:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1844813824 bytes 2024-09-16T23:18:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1856560128 bytes 2024-09-16T23:18:13.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1932780544 bytes 2024-09-16T23:18:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1949171712 bytes 2024-09-16T23:18:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1979565056 bytes 2024-09-16T23:18:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2022775808 bytes 2024-09-16T23:18:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2107031552 bytes 2024-09-16T23:18:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2115097600 bytes 2024-09-16T23:18:14.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2195112960 bytes 2024-09-16T23:18:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2257884160 bytes 2024-09-16T23:18:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2265696256 bytes 2024-09-16T23:18:14.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2316414976 bytes 2024-09-16T23:18:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2376029184 bytes 2024-09-16T23:18:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2413820928 bytes 2024-09-16T23:18:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2486283264 bytes 2024-09-16T23:18:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2531333120 bytes 2024-09-16T23:18:15.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2572979200 bytes 2024-09-16T23:18:15.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2637523968 bytes 2024-09-16T23:18:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2689907712 bytes 2024-09-16T23:18:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2759803904 bytes 2024-09-16T23:18:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2841304064 bytes 2024-09-16T23:18:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2864067584 bytes 2024-09-16T23:18:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2972865536 bytes 2024-09-16T23:18:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3009958912 bytes 2024-09-16T23:18:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3023744000 bytes 2024-09-16T23:18:16.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3064999936 bytes 2024-09-16T23:18:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3089328128 bytes 2024-09-16T23:18:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3158095872 bytes 2024-09-16T23:18:16.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3197029376 bytes 2024-09-16T23:18:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3243597824 bytes 2024-09-16T23:18:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3244219392 bytes 2024-09-16T23:18:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3284869120 bytes 2024-09-16T23:18:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3327818752 bytes 2024-09-16T23:18:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3363203072 bytes 2024-09-16T23:18:17.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3382492160 bytes 2024-09-16T23:18:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3438990336 bytes 2024-09-16T23:18:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3439591424 bytes 2024-09-16T23:18:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3490658304 bytes 2024-09-16T23:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3538967552 bytes 2024-09-16T23:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3586567168 bytes 2024-09-16T23:18:17.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3688163328 bytes 2024-09-16T23:18:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3745661952 bytes 2024-09-16T23:18:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3785152512 bytes 2024-09-16T23:18:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3834397696 bytes 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_5 2024-09-16T23:18:18.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:18.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:18.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:18:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T23:18:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_5': 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12a366e0871b 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12a366e0871b 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:18 2024 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:18 2024 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:18 2024 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: f84d3a0f-2e8c-472f-b6aa-47f1ebb35940 2024-09-16T23:18:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:18:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=5 2024-09-16T23:18:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 72537088 bytes 2024-09-16T23:18:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 92752896 bytes 2024-09-16T23:18:19.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 174379008 bytes 2024-09-16T23:18:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 244626432 bytes 2024-09-16T23:18:19.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 290312192 bytes 2024-09-16T23:18:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 334057472 bytes 2024-09-16T23:18:19.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 409197568 bytes 2024-09-16T23:18:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 449132544 bytes 2024-09-16T23:18:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 487272448 bytes 2024-09-16T23:18:20.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 541778944 bytes 2024-09-16T23:18:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 596515840 bytes 2024-09-16T23:18:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 629718016 bytes 2024-09-16T23:18:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 716967936 bytes 2024-09-16T23:18:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 767994880 bytes 2024-09-16T23:18:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 768602112 bytes 2024-09-16T23:18:20.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 809592832 bytes 2024-09-16T23:18:21.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 881013760 bytes 2024-09-16T23:18:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 910880768 bytes 2024-09-16T23:18:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 976490496 bytes 2024-09-16T23:18:21.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1034878976 bytes 2024-09-16T23:18:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1083273216 bytes 2024-09-16T23:18:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1086557184 bytes 2024-09-16T23:18:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1114287104 bytes 2024-09-16T23:18:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1177846784 bytes 2024-09-16T23:18:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1191372800 bytes 2024-09-16T23:18:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1219756032 bytes 2024-09-16T23:18:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1289072640 bytes 2024-09-16T23:18:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1340438528 bytes 2024-09-16T23:18:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1390131200 bytes 2024-09-16T23:18:22.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1390750720 bytes 2024-09-16T23:18:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1458942976 bytes 2024-09-16T23:18:22.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1516652544 bytes 2024-09-16T23:18:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1560976384 bytes 2024-09-16T23:18:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1599128576 bytes 2024-09-16T23:18:23.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1665928192 bytes 2024-09-16T23:18:23.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1752947712 bytes 2024-09-16T23:18:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1773936640 bytes 2024-09-16T23:18:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1806679040 bytes 2024-09-16T23:18:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1836007424 bytes 2024-09-16T23:18:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1871376384 bytes 2024-09-16T23:18:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1900197888 bytes 2024-09-16T23:18:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1920416768 bytes 2024-09-16T23:18:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1928336384 bytes 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_6 2024-09-16T23:18:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:18:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2024-09-16T23:18:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:24.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-16T23:18:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:18:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T23:18:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_6': 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12af9c1c8582 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12af9c1c8582 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:24 2024 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:24 2024 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:24 2024 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c94c7221-b8a2-4e38-a50c-e34a1f45df1c 2024-09-16T23:18:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:18:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=5 2024-09-16T23:18:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 15419392 bytes 2024-09-16T23:18:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 41216000 bytes 2024-09-16T23:18:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 84984832 bytes 2024-09-16T23:18:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 133947392 bytes 2024-09-16T23:18:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 190866432 bytes 2024-09-16T23:18:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 288741376 bytes 2024-09-16T23:18:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 335451136 bytes 2024-09-16T23:18:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 383717376 bytes 2024-09-16T23:18:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 411620352 bytes 2024-09-16T23:18:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 422756352 bytes 2024-09-16T23:18:26.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 441090048 bytes 2024-09-16T23:18:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 497755136 bytes 2024-09-16T23:18:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 539937792 bytes 2024-09-16T23:18:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 629238784 bytes 2024-09-16T23:18:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 707214336 bytes 2024-09-16T23:18:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 709526528 bytes 2024-09-16T23:18:27.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 728496128 bytes 2024-09-16T23:18:27.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 822308864 bytes 2024-09-16T23:18:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 924767232 bytes 2024-09-16T23:18:27.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 980792320 bytes 2024-09-16T23:18:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1050095616 bytes 2024-09-16T23:18:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1067152384 bytes 2024-09-16T23:18:27.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1139860480 bytes 2024-09-16T23:18:27.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1177554944 bytes 2024-09-16T23:18:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1192279040 bytes 2024-09-16T23:18:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1192901632 bytes 2024-09-16T23:18:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1261190144 bytes 2024-09-16T23:18:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1294212096 bytes 2024-09-16T23:18:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1319048192 bytes 2024-09-16T23:18:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1342606336 bytes 2024-09-16T23:18:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1401202688 bytes 2024-09-16T23:18:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1463877632 bytes 2024-09-16T23:18:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1508807680 bytes 2024-09-16T23:18:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1542072320 bytes 2024-09-16T23:18:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1543278592 bytes 2024-09-16T23:18:29.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1645101056 bytes 2024-09-16T23:18:29.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1725022208 bytes 2024-09-16T23:18:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1754810368 bytes 2024-09-16T23:18:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1826375680 bytes 2024-09-16T23:18:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1876741120 bytes 2024-09-16T23:18:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1948095488 bytes 2024-09-16T23:18:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1967949824 bytes 2024-09-16T23:18:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:30.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_7 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-16T23:18:30.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:30.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:30.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T23:18:30.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T23:18:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_7': 2024-09-16T23:18:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:31.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12ba69348f4e 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12ba69348f4e 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:30 2024 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:30 2024 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:30 2024 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8e4da14b-dfc0-4158-b26e-7af55852456e 2024-09-16T23:18:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:18:31.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:18:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 30047232 bytes 2024-09-16T23:18:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 53277696 bytes 2024-09-16T23:18:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 103191552 bytes 2024-09-16T23:18:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 144037888 bytes 2024-09-16T23:18:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 257285120 bytes 2024-09-16T23:18:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 265719808 bytes 2024-09-16T23:18:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 266315776 bytes 2024-09-16T23:18:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 296146944 bytes 2024-09-16T23:18:32.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 335386624 bytes 2024-09-16T23:18:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 383016960 bytes 2024-09-16T23:18:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 428273664 bytes 2024-09-16T23:18:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 473791488 bytes 2024-09-16T23:18:32.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 558983168 bytes 2024-09-16T23:18:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 640198656 bytes 2024-09-16T23:18:32.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 678700032 bytes 2024-09-16T23:18:32.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 699504640 bytes 2024-09-16T23:18:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 714205184 bytes 2024-09-16T23:18:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 767480832 bytes 2024-09-16T23:18:33.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 785973248 bytes 2024-09-16T23:18:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 801745920 bytes 2024-09-16T23:18:33.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 813608960 bytes 2024-09-16T23:18:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 863861760 bytes 2024-09-16T23:18:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 873908224 bytes 2024-09-16T23:18:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 884929536 bytes 2024-09-16T23:18:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 931437568 bytes 2024-09-16T23:18:33.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 957178880 bytes 2024-09-16T23:18:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 997091328 bytes 2024-09-16T23:18:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1049540608 bytes 2024-09-16T23:18:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1098166272 bytes 2024-09-16T23:18:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1128397824 bytes 2024-09-16T23:18:34.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1178134528 bytes 2024-09-16T23:18:34.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1199285248 bytes 2024-09-16T23:18:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1218231296 bytes 2024-09-16T23:18:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1263053824 bytes 2024-09-16T23:18:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1338237952 bytes 2024-09-16T23:18:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1361494016 bytes 2024-09-16T23:18:35.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1381421056 bytes 2024-09-16T23:18:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1432453120 bytes 2024-09-16T23:18:35.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1454992384 bytes 2024-09-16T23:18:35.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1490048000 bytes 2024-09-16T23:18:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1514586112 bytes 2024-09-16T23:18:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1576189952 bytes 2024-09-16T23:18:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1646811136 bytes 2024-09-16T23:18:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1712718848 bytes 2024-09-16T23:18:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1734181888 bytes 2024-09-16T23:18:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1782190080 bytes 2024-09-16T23:18:36.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1814706176 bytes 2024-09-16T23:18:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1832329216 bytes 2024-09-16T23:18:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1866649600 bytes 2024-09-16T23:18:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1911425024 bytes 2024-09-16T23:18:36.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1935709184 bytes 2024-09-16T23:18:36.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 1953160192 bytes 2024-09-16T23:18:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2003395584 bytes 2024-09-16T23:18:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2053873664 bytes 2024-09-16T23:18:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2098954240 bytes 2024-09-16T23:18:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2138145792 bytes 2024-09-16T23:18:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2205257728 bytes 2024-09-16T23:18:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2260063232 bytes 2024-09-16T23:18:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2327475200 bytes 2024-09-16T23:18:37.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2356532224 bytes 2024-09-16T23:18:37.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2411926528 bytes 2024-09-16T23:18:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2426543104 bytes 2024-09-16T23:18:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2477061120 bytes 2024-09-16T23:18:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2539252736 bytes 2024-09-16T23:18:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2559093760 bytes 2024-09-16T23:18:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2593982464 bytes 2024-09-16T23:18:38.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2667357184 bytes 2024-09-16T23:18:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2705104896 bytes 2024-09-16T23:18:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2763390976 bytes 2024-09-16T23:18:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2769482752 bytes 2024-09-16T23:18:39.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2849471488 bytes 2024-09-16T23:18:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2902635520 bytes 2024-09-16T23:18:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2907174912 bytes 2024-09-16T23:18:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2926965760 bytes 2024-09-16T23:18:39.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2980813824 bytes 2024-09-16T23:18:39.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2981413888 bytes 2024-09-16T23:18:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3016583168 bytes 2024-09-16T23:18:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3111789568 bytes 2024-09-16T23:18:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3151862784 bytes 2024-09-16T23:18:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3185558528 bytes 2024-09-16T23:18:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3211941888 bytes 2024-09-16T23:18:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3239843840 bytes 2024-09-16T23:18:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3273931776 bytes 2024-09-16T23:18:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3302887424 bytes 2024-09-16T23:18:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3321661440 bytes 2024-09-16T23:18:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3353277440 bytes 2024-09-16T23:18:40.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3368778752 bytes 2024-09-16T23:18:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3449931776 bytes 2024-09-16T23:18:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3472962560 bytes 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_8 2024-09-16T23:18:41.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:41.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:41.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:18:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T23:18:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_8': 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12c66505df15 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12c66505df15 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:41 2024 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:41 2024 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:41 2024 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: bd37f003-9696-4966-8734-3bc10187a7c9 2024-09-16T23:18:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:18:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:18:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:42.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 25912320 bytes 2024-09-16T23:18:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 80059392 bytes 2024-09-16T23:18:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 125301760 bytes 2024-09-16T23:18:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 163089408 bytes 2024-09-16T23:18:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 190021632 bytes 2024-09-16T23:18:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 214841344 bytes 2024-09-16T23:18:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 248723456 bytes 2024-09-16T23:18:42.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 290259968 bytes 2024-09-16T23:18:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 369746944 bytes 2024-09-16T23:18:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 405627904 bytes 2024-09-16T23:18:43.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 424665088 bytes 2024-09-16T23:18:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 425304064 bytes 2024-09-16T23:18:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 452397056 bytes 2024-09-16T23:18:43.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 462036992 bytes 2024-09-16T23:18:43.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 516697088 bytes 2024-09-16T23:18:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 537069568 bytes 2024-09-16T23:18:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 571134976 bytes 2024-09-16T23:18:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 596627456 bytes 2024-09-16T23:18:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 617425920 bytes 2024-09-16T23:18:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 662695936 bytes 2024-09-16T23:18:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 667994112 bytes 2024-09-16T23:18:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 704654336 bytes 2024-09-16T23:18:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 736485376 bytes 2024-09-16T23:18:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 804678656 bytes 2024-09-16T23:18:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 826564608 bytes 2024-09-16T23:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 902220800 bytes 2024-09-16T23:18:45.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 947364864 bytes 2024-09-16T23:18:45.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1028446208 bytes 2024-09-16T23:18:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1136265216 bytes 2024-09-16T23:18:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1227060224 bytes 2024-09-16T23:18:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1256909824 bytes 2024-09-16T23:18:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1306565632 bytes 2024-09-16T23:18:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1338411008 bytes 2024-09-16T23:18:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1353084928 bytes 2024-09-16T23:18:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1396571136 bytes 2024-09-16T23:18:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1451543552 bytes 2024-09-16T23:18:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1506091008 bytes 2024-09-16T23:18:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1539776512 bytes 2024-09-16T23:18:46.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1577804800 bytes 2024-09-16T23:18:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1647639552 bytes 2024-09-16T23:18:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1655681024 bytes 2024-09-16T23:18:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1754490880 bytes 2024-09-16T23:18:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1816057856 bytes 2024-09-16T23:18:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1829287936 bytes 2024-09-16T23:18:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1898968064 bytes 2024-09-16T23:18:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1964608512 bytes 2024-09-16T23:18:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2006348800 bytes 2024-09-16T23:18:47.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2016040960 bytes 2024-09-16T23:18:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2028328960 bytes 2024-09-16T23:18:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2130864128 bytes 2024-09-16T23:18:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2234779648 bytes 2024-09-16T23:18:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2328056832 bytes 2024-09-16T23:18:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2351053824 bytes 2024-09-16T23:18:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2390133760 bytes 2024-09-16T23:18:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2390749184 bytes 2024-09-16T23:18:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2391359488 bytes 2024-09-16T23:18:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2473973760 bytes 2024-09-16T23:18:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2536185856 bytes 2024-09-16T23:18:48.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2557458432 bytes 2024-09-16T23:18:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2608442368 bytes 2024-09-16T23:18:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2654337024 bytes 2024-09-16T23:18:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2689427456 bytes 2024-09-16T23:18:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2699515904 bytes 2024-09-16T23:18:49.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2729673728 bytes 2024-09-16T23:18:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2801165312 bytes 2024-09-16T23:18:49.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2875686912 bytes 2024-09-16T23:18:49.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2907205632 bytes 2024-09-16T23:18:49.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2959253504 bytes 2024-09-16T23:18:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3008141312 bytes 2024-09-16T23:18:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3035534336 bytes 2024-09-16T23:18:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3091678208 bytes 2024-09-16T23:18:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3114528768 bytes 2024-09-16T23:18:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3169929216 bytes 2024-09-16T23:18:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3209367552 bytes 2024-09-16T23:18:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3246242816 bytes 2024-09-16T23:18:50.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3258469376 bytes 2024-09-16T23:18:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3285359616 bytes 2024-09-16T23:18:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3315136512 bytes 2024-09-16T23:18:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3421295616 bytes 2024-09-16T23:18:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3444646912 bytes 2024-09-16T23:18:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3479132160 bytes 2024-09-16T23:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3522078720 bytes 2024-09-16T23:18:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3552086016 bytes 2024-09-16T23:18:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3620882432 bytes 2024-09-16T23:18:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3676821504 bytes 2024-09-16T23:18:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3736144896 bytes 2024-09-16T23:18:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3782986752 bytes 2024-09-16T23:18:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3817584640 bytes 2024-09-16T23:18:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3831235584 bytes 2024-09-16T23:18:52.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3917704192 bytes 2024-09-16T23:18:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3942304768 bytes 2024-09-16T23:18:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3960172544 bytes 2024-09-16T23:18:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3960782848 bytes 2024-09-16T23:18:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4018824192 bytes 2024-09-16T23:18:53.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4086720512 bytes 2024-09-16T23:18:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4141223936 bytes 2024-09-16T23:18:53.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4143663104 bytes 2024-09-16T23:18:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4159278080 bytes 2024-09-16T23:18:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4159881216 bytes 2024-09-16T23:18:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4193128448 bytes 2024-09-16T23:18:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4227747840 bytes 2024-09-16T23:18:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4274722816 bytes 2024-09-16T23:18:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4350280704 bytes 2024-09-16T23:18:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4439437312 bytes 2024-09-16T23:18:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4464868352 bytes 2024-09-16T23:18:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4508941312 bytes 2024-09-16T23:18:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4563321856 bytes 2024-09-16T23:18:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4580865024 bytes 2024-09-16T23:18:54.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4595138560 bytes 2024-09-16T23:18:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4663846912 bytes 2024-09-16T23:18:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4676265984 bytes 2024-09-16T23:18:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4705441792 bytes 2024-09-16T23:18:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4742501376 bytes 2024-09-16T23:18:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4830217216 bytes 2024-09-16T23:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11500 IOs, 4856163328 bytes 2024-09-16T23:18:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11600 IOs, 4902797312 bytes 2024-09-16T23:18:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11700 IOs, 4928424960 bytes 2024-09-16T23:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11800 IOs, 4997420032 bytes 2024-09-16T23:18:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11900 IOs, 5035052032 bytes 2024-09-16T23:18:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12000 IOs, 5061732352 bytes 2024-09-16T23:18:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:12100 IOs, 5097410560 bytes 2024-09-16T23:18:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:18:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:18:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:18:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_9 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:18:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2024-09-16T23:18:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:18:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:18:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T23:18:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:18:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_9': 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12d2be4b638e 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12d2be4b638e 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:56 2024 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:56 2024 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:56 2024 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 89e925c9-d5c8-42fb-9de9-72a27fc3d8b5 2024-09-16T23:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:18:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:18:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T23:18:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:18:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:18:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:18:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:18:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=11 2024-09-16T23:18:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:18:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:18:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 19610624 bytes 2024-09-16T23:18:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 67853312 bytes 2024-09-16T23:18:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 94737408 bytes 2024-09-16T23:18:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 144629760 bytes 2024-09-16T23:18:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 171193344 bytes 2024-09-16T23:18:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 284199936 bytes 2024-09-16T23:18:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 328590336 bytes 2024-09-16T23:18:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 360730624 bytes 2024-09-16T23:18:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 377464832 bytes 2024-09-16T23:18:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 403132416 bytes 2024-09-16T23:18:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 404612096 bytes 2024-09-16T23:18:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 483128320 bytes 2024-09-16T23:18:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 496821248 bytes 2024-09-16T23:18:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 560335872 bytes 2024-09-16T23:18:58.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 642995200 bytes 2024-09-16T23:18:58.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 703961088 bytes 2024-09-16T23:18:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 748494848 bytes 2024-09-16T23:18:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 785297408 bytes 2024-09-16T23:18:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 816250880 bytes 2024-09-16T23:18:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 916177920 bytes 2024-09-16T23:18:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 937593856 bytes 2024-09-16T23:18:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 982174720 bytes 2024-09-16T23:18:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1021446144 bytes 2024-09-16T23:18:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1076049920 bytes 2024-09-16T23:18:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1153420288 bytes 2024-09-16T23:18:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1190492160 bytes 2024-09-16T23:18:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1239520256 bytes 2024-09-16T23:18:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1284468736 bytes 2024-09-16T23:18:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1285076992 bytes 2024-09-16T23:19:00.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1335627776 bytes 2024-09-16T23:19:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1432326144 bytes 2024-09-16T23:19:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1456856064 bytes 2024-09-16T23:19:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1521021952 bytes 2024-09-16T23:19:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1613888512 bytes 2024-09-16T23:19:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1694226432 bytes 2024-09-16T23:19:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1770054656 bytes 2024-09-16T23:19:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1787062272 bytes 2024-09-16T23:19:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1815804928 bytes 2024-09-16T23:19:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1874355200 bytes 2024-09-16T23:19:01.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1915414528 bytes 2024-09-16T23:19:01.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1930795008 bytes 2024-09-16T23:19:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1963424768 bytes 2024-09-16T23:19:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2014744576 bytes 2024-09-16T23:19:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2128710656 bytes 2024-09-16T23:19:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2158015488 bytes 2024-09-16T23:19:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2181848064 bytes 2024-09-16T23:19:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2300123136 bytes 2024-09-16T23:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2351578112 bytes 2024-09-16T23:19:02.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2395335680 bytes 2024-09-16T23:19:02.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2469526528 bytes 2024-09-16T23:19:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2523345920 bytes 2024-09-16T23:19:02.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2559026176 bytes 2024-09-16T23:19:02.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2585208832 bytes 2024-09-16T23:19:03.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2596685824 bytes 2024-09-16T23:19:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2673475584 bytes 2024-09-16T23:19:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2717916160 bytes 2024-09-16T23:19:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2772837376 bytes 2024-09-16T23:19:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2773442560 bytes 2024-09-16T23:19:03.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2798221312 bytes 2024-09-16T23:19:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2804343808 bytes 2024-09-16T23:19:03.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2866157568 bytes 2024-09-16T23:19:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2866782208 bytes 2024-09-16T23:19:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2898075648 bytes 2024-09-16T23:19:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2943073280 bytes 2024-09-16T23:19:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2993673216 bytes 2024-09-16T23:19:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2998405120 bytes 2024-09-16T23:19:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3038317568 bytes 2024-09-16T23:19:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3095363584 bytes 2024-09-16T23:19:04.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3137883136 bytes 2024-09-16T23:19:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3201729536 bytes 2024-09-16T23:19:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3229154304 bytes 2024-09-16T23:19:05.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3240066048 bytes 2024-09-16T23:19:05.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3247379456 bytes 2024-09-16T23:19:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3248010240 bytes 2024-09-16T23:19:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3273835520 bytes 2024-09-16T23:19:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3313317888 bytes 2024-09-16T23:19:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3339063296 bytes 2024-09-16T23:19:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3385003008 bytes 2024-09-16T23:19:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3436217344 bytes 2024-09-16T23:19:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3484691456 bytes 2024-09-16T23:19:06.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3506836480 bytes 2024-09-16T23:19:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3567357952 bytes 2024-09-16T23:19:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3604664320 bytes 2024-09-16T23:19:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3621118976 bytes 2024-09-16T23:19:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3722187776 bytes 2024-09-16T23:19:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3784084480 bytes 2024-09-16T23:19:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3814434816 bytes 2024-09-16T23:19:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3839822848 bytes 2024-09-16T23:19:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3908758528 bytes 2024-09-16T23:19:07.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3943891968 bytes 2024-09-16T23:19:07.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3960475648 bytes 2024-09-16T23:19:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4000970752 bytes 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_10 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2024-09-16T23:19:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:07.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:07.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T23:19:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T23:19:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_10': 2024-09-16T23:19:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12de313eb368 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12de313eb368 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:08.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:07 2024 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:07 2024 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:07 2024 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:08.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ec61c977-0c0f-41e7-8a23-423f6faad7dc 2024-09-16T23:19:08.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T23:19:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:19:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 74125312 bytes 2024-09-16T23:19:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 145763328 bytes 2024-09-16T23:19:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 162586624 bytes 2024-09-16T23:19:09.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 240478208 bytes 2024-09-16T23:19:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 314457088 bytes 2024-09-16T23:19:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 358905856 bytes 2024-09-16T23:19:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 392976384 bytes 2024-09-16T23:19:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 467193856 bytes 2024-09-16T23:19:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 476833792 bytes 2024-09-16T23:19:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 482483200 bytes 2024-09-16T23:19:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 494002176 bytes 2024-09-16T23:19:10.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 494771200 bytes 2024-09-16T23:19:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 565745664 bytes 2024-09-16T23:19:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 576696320 bytes 2024-09-16T23:19:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 634819584 bytes 2024-09-16T23:19:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 707353600 bytes 2024-09-16T23:19:10.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 735447040 bytes 2024-09-16T23:19:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 758006784 bytes 2024-09-16T23:19:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 817999872 bytes 2024-09-16T23:19:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 862617600 bytes 2024-09-16T23:19:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 871422976 bytes 2024-09-16T23:19:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 918922240 bytes 2024-09-16T23:19:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 938556416 bytes 2024-09-16T23:19:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1015160832 bytes 2024-09-16T23:19:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1035127808 bytes 2024-09-16T23:19:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1057074176 bytes 2024-09-16T23:19:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1156117504 bytes 2024-09-16T23:19:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1195927552 bytes 2024-09-16T23:19:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1237402624 bytes 2024-09-16T23:19:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1295026176 bytes 2024-09-16T23:19:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1341031424 bytes 2024-09-16T23:19:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1423219712 bytes 2024-09-16T23:19:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1469217792 bytes 2024-09-16T23:19:12.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1484991488 bytes 2024-09-16T23:19:12.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1524217856 bytes 2024-09-16T23:19:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1535373312 bytes 2024-09-16T23:19:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1618388992 bytes 2024-09-16T23:19:13.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1728068608 bytes 2024-09-16T23:19:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1755684864 bytes 2024-09-16T23:19:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1801332736 bytes 2024-09-16T23:19:13.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1936523264 bytes 2024-09-16T23:19:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1937134592 bytes 2024-09-16T23:19:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1949129728 bytes 2024-09-16T23:19:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1972676608 bytes 2024-09-16T23:19:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1993416704 bytes 2024-09-16T23:19:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2036125696 bytes 2024-09-16T23:19:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2099077120 bytes 2024-09-16T23:19:14.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2154487808 bytes 2024-09-16T23:19:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2192423936 bytes 2024-09-16T23:19:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2246850560 bytes 2024-09-16T23:19:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2298067968 bytes 2024-09-16T23:19:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2366344192 bytes 2024-09-16T23:19:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2419046400 bytes 2024-09-16T23:19:15.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2469188608 bytes 2024-09-16T23:19:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2501335040 bytes 2024-09-16T23:19:15.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2555581440 bytes 2024-09-16T23:19:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2584534016 bytes 2024-09-16T23:19:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2641480704 bytes 2024-09-16T23:19:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2696197120 bytes 2024-09-16T23:19:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2756089856 bytes 2024-09-16T23:19:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2772499456 bytes 2024-09-16T23:19:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2794669056 bytes 2024-09-16T23:19:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2862072832 bytes 2024-09-16T23:19:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2880188416 bytes 2024-09-16T23:19:16.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2920624128 bytes 2024-09-16T23:19:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3011421184 bytes 2024-09-16T23:19:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3042776064 bytes 2024-09-16T23:19:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3112436736 bytes 2024-09-16T23:19:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3134440448 bytes 2024-09-16T23:19:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3194514432 bytes 2024-09-16T23:19:17.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3272841216 bytes 2024-09-16T23:19:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3325603840 bytes 2024-09-16T23:19:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3413933056 bytes 2024-09-16T23:19:17.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3434833920 bytes 2024-09-16T23:19:17.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3503069184 bytes 2024-09-16T23:19:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3565483008 bytes 2024-09-16T23:19:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3581256704 bytes 2024-09-16T23:19:18.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3590060032 bytes 2024-09-16T23:19:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3678005248 bytes 2024-09-16T23:19:18.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.38132423078138167) > weight (0.25) 2024-09-16T23:19:18.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5983143422321313) > weight (0.25) 2024-09-16T23:19:18.348 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.948013761879685) > weight (0.25) 2024-09-16T23:19:18.348 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-16T23:19:18.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-16T23:19:18.349 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:19:18.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:19:18.351+0000 7f91fafe9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40769) UID: 1000 2024-09-16T23:19:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3679924224 bytes 2024-09-16T23:19:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3680528384 bytes 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_11 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2024-09-16T23:19:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-16T23:19:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:19.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-16T23:19:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_11': 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 12eaa48f9cc2 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.12eaa48f9cc2 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:18 2024 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:18 2024 2024-09-16T23:19:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:18 2024 2024-09-16T23:19:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 268e0ae9-2b9e-4a82-8b83-58d27f2a3cd5 2024-09-16T23:19:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:19:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:19:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:19.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 36875264 bytes 2024-09-16T23:19:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 77786112 bytes 2024-09-16T23:19:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 140992512 bytes 2024-09-16T23:19:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 167321600 bytes 2024-09-16T23:19:20.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 259439616 bytes 2024-09-16T23:19:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 322765824 bytes 2024-09-16T23:19:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 391635968 bytes 2024-09-16T23:19:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 426634240 bytes 2024-09-16T23:19:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 444409856 bytes 2024-09-16T23:19:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 449182720 bytes 2024-09-16T23:19:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 488081408 bytes 2024-09-16T23:19:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 521836544 bytes 2024-09-16T23:19:21.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 577842176 bytes 2024-09-16T23:19:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 641908736 bytes 2024-09-16T23:19:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 651603968 bytes 2024-09-16T23:19:21.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 705354752 bytes 2024-09-16T23:19:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 705969152 bytes 2024-09-16T23:19:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 718530560 bytes 2024-09-16T23:19:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 727830528 bytes 2024-09-16T23:19:21.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 736215040 bytes 2024-09-16T23:19:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 794862592 bytes 2024-09-16T23:19:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 872402944 bytes 2024-09-16T23:19:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 886801408 bytes 2024-09-16T23:19:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 923216896 bytes 2024-09-16T23:19:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 923845632 bytes 2024-09-16T23:19:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 974135296 bytes 2024-09-16T23:19:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1063415808 bytes 2024-09-16T23:19:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1102073856 bytes 2024-09-16T23:19:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1199901696 bytes 2024-09-16T23:19:23.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1229434880 bytes 2024-09-16T23:19:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1300045824 bytes 2024-09-16T23:19:23.350 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-16T23:19:23.350 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:19:23.351 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-16T23:19:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-16T23:19:23.351 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-16T23:19:23.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-16T23:19:23.355 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-16T23:19:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1347233792 bytes 2024-09-16T23:19:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1429814272 bytes 2024-09-16T23:19:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1457362944 bytes 2024-09-16T23:19:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1503190016 bytes 2024-09-16T23:19:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1607512064 bytes 2024-09-16T23:19:24.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1649983488 bytes 2024-09-16T23:19:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1706166272 bytes 2024-09-16T23:19:24.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1768979456 bytes 2024-09-16T23:19:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1805213696 bytes 2024-09-16T23:19:24.531 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:19:24.532+0000 7fbcef505640 -1 reset not still connected to 0x56517d1d04e0 2024-09-16T23:19:24.534 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:19:24.535+0000 7fbcef505640 -1 reset not still connected to 0x56517f4fa340 2024-09-16T23:19:24.534 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:19:24.536+0000 7fbcef505640 -1 reset not still connected to 0x565180d9f1e0 2024-09-16T23:19:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1842843648 bytes 2024-09-16T23:19:24.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1911910400 bytes 2024-09-16T23:19:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1924711424 bytes 2024-09-16T23:19:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1947190272 bytes 2024-09-16T23:19:25.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1947824128 bytes 2024-09-16T23:19:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2004137984 bytes 2024-09-16T23:19:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2061561856 bytes 2024-09-16T23:19:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2104303616 bytes 2024-09-16T23:19:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2228097024 bytes 2024-09-16T23:19:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2244806656 bytes 2024-09-16T23:19:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2258494464 bytes 2024-09-16T23:19:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2294221824 bytes 2024-09-16T23:19:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2333965312 bytes 2024-09-16T23:19:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2397589504 bytes 2024-09-16T23:19:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2474921984 bytes 2024-09-16T23:19:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2475530240 bytes 2024-09-16T23:19:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2496457728 bytes 2024-09-16T23:19:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2560986112 bytes 2024-09-16T23:19:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2561605632 bytes 2024-09-16T23:19:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2585332736 bytes 2024-09-16T23:19:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2615520256 bytes 2024-09-16T23:19:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2643194880 bytes 2024-09-16T23:19:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2720580608 bytes 2024-09-16T23:19:27.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2748332032 bytes 2024-09-16T23:19:27.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2784145408 bytes 2024-09-16T23:19:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2811997184 bytes 2024-09-16T23:19:27.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2866592768 bytes 2024-09-16T23:19:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2883409920 bytes 2024-09-16T23:19:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2983997440 bytes 2024-09-16T23:19:27.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3007792128 bytes 2024-09-16T23:19:28.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3015166976 bytes 2024-09-16T23:19:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3064240128 bytes 2024-09-16T23:19:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3089945600 bytes 2024-09-16T23:19:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3111524352 bytes 2024-09-16T23:19:28.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3160864768 bytes 2024-09-16T23:19:28.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3187248128 bytes 2024-09-16T23:19:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3232429056 bytes 2024-09-16T23:19:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3279459328 bytes 2024-09-16T23:19:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3297773568 bytes 2024-09-16T23:19:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3323319296 bytes 2024-09-16T23:19:29.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3378660352 bytes 2024-09-16T23:19:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3426246656 bytes 2024-09-16T23:19:29.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3479831552 bytes 2024-09-16T23:19:29.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3546611712 bytes 2024-09-16T23:19:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3576014848 bytes 2024-09-16T23:19:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3614950400 bytes 2024-09-16T23:19:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3640303616 bytes 2024-09-16T23:19:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3695247360 bytes 2024-09-16T23:19:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3749438464 bytes 2024-09-16T23:19:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3839941632 bytes 2024-09-16T23:19:30.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3854106624 bytes 2024-09-16T23:19:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3918098432 bytes 2024-09-16T23:19:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4010943488 bytes 2024-09-16T23:19:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4064979968 bytes 2024-09-16T23:19:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4135499776 bytes 2024-09-16T23:19:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4210302976 bytes 2024-09-16T23:19:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4212631552 bytes 2024-09-16T23:19:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4238168064 bytes 2024-09-16T23:19:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4294047744 bytes 2024-09-16T23:19:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4324388864 bytes 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_12 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:31.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-16T23:19:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_12': 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1300b20f0355 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1300b20f0355 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:31 2024 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:31 2024 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:31 2024 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b9122161-1c25-422e-ae7b-083efda568a9 2024-09-16T23:19:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-16T23:19:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:19:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=5 2024-09-16T23:19:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:32.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:32.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 56993792 bytes 2024-09-16T23:19:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 107169792 bytes 2024-09-16T23:19:33.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 134211584 bytes 2024-09-16T23:19:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 214223872 bytes 2024-09-16T23:19:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 259639296 bytes 2024-09-16T23:19:33.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 325044224 bytes 2024-09-16T23:19:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 356721664 bytes 2024-09-16T23:19:33.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 453668864 bytes 2024-09-16T23:19:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 490718208 bytes 2024-09-16T23:19:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 491346944 bytes 2024-09-16T23:19:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 546305024 bytes 2024-09-16T23:19:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 582583296 bytes 2024-09-16T23:19:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 628359168 bytes 2024-09-16T23:19:34.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 667253760 bytes 2024-09-16T23:19:34.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 722891776 bytes 2024-09-16T23:19:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 756325376 bytes 2024-09-16T23:19:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 769408000 bytes 2024-09-16T23:19:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 837950464 bytes 2024-09-16T23:19:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 858587136 bytes 2024-09-16T23:19:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 871712768 bytes 2024-09-16T23:19:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 914795520 bytes 2024-09-16T23:19:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 932106240 bytes 2024-09-16T23:19:35.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 983714816 bytes 2024-09-16T23:19:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1025253376 bytes 2024-09-16T23:19:35.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1035787264 bytes 2024-09-16T23:19:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1112772608 bytes 2024-09-16T23:19:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1129403392 bytes 2024-09-16T23:19:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1200927744 bytes 2024-09-16T23:19:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1215617024 bytes 2024-09-16T23:19:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1272549376 bytes 2024-09-16T23:19:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1297320960 bytes 2024-09-16T23:19:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1308830720 bytes 2024-09-16T23:19:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1343594496 bytes 2024-09-16T23:19:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1428809728 bytes 2024-09-16T23:19:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1486999552 bytes 2024-09-16T23:19:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1536507904 bytes 2024-09-16T23:19:37.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1591485440 bytes 2024-09-16T23:19:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1658673152 bytes 2024-09-16T23:19:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1715672064 bytes 2024-09-16T23:19:37.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1718808576 bytes 2024-09-16T23:19:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1762914304 bytes 2024-09-16T23:19:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1788633088 bytes 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_13 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2024-09-16T23:19:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-16T23:19:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:19:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-16T23:19:38.445 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6616098893440042) > weight (0.25) 2024-09-16T23:19:38.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7474397080974648) > weight (0.25) 2024-09-16T23:19:38.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9710521553008468) > weight (0.25) 2024-09-16T23:19:38.446 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8315067783320516) > weight (0.25) 2024-09-16T23:19:38.446 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2024-09-16T23:19:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:38.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_13': 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 130cd29816ca 2024-09-16T23:19:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.130cd29816ca 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:37 2024 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:37 2024 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:37 2024 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b92b2830-e993-4cb1-800e-5126fdb3887f 2024-09-16T23:19:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-16T23:19:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:38.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:19:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:19:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:38.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 23879680 bytes 2024-09-16T23:19:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 24505344 bytes 2024-09-16T23:19:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 74726400 bytes 2024-09-16T23:19:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 138584064 bytes 2024-09-16T23:19:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 207887360 bytes 2024-09-16T23:19:39.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 234491904 bytes 2024-09-16T23:19:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 259217408 bytes 2024-09-16T23:19:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 310555648 bytes 2024-09-16T23:19:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 372840448 bytes 2024-09-16T23:19:40.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 462361600 bytes 2024-09-16T23:19:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 471564288 bytes 2024-09-16T23:19:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 491915264 bytes 2024-09-16T23:19:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 533881856 bytes 2024-09-16T23:19:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 551348224 bytes 2024-09-16T23:19:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 605049856 bytes 2024-09-16T23:19:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 605631488 bytes 2024-09-16T23:19:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 624787456 bytes 2024-09-16T23:19:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 717714432 bytes 2024-09-16T23:19:41.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 718323712 bytes 2024-09-16T23:19:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 811846656 bytes 2024-09-16T23:19:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 812469248 bytes 2024-09-16T23:19:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 853392384 bytes 2024-09-16T23:19:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 944428032 bytes 2024-09-16T23:19:41.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 998148096 bytes 2024-09-16T23:19:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1017788416 bytes 2024-09-16T23:19:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1079302144 bytes 2024-09-16T23:19:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1155768320 bytes 2024-09-16T23:19:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1192520704 bytes 2024-09-16T23:19:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1293400064 bytes 2024-09-16T23:19:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1387513856 bytes 2024-09-16T23:19:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1439905792 bytes 2024-09-16T23:19:42.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1491858432 bytes 2024-09-16T23:19:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1570065408 bytes 2024-09-16T23:19:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1571520512 bytes 2024-09-16T23:19:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1608888320 bytes 2024-09-16T23:19:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1655261184 bytes 2024-09-16T23:19:43.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1685640192 bytes 2024-09-16T23:19:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1767385088 bytes 2024-09-16T23:19:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1785573376 bytes 2024-09-16T23:19:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1830793216 bytes 2024-09-16T23:19:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1957178368 bytes 2024-09-16T23:19:44.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1997856768 bytes 2024-09-16T23:19:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2116220928 bytes 2024-09-16T23:19:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2143611904 bytes 2024-09-16T23:19:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2157617152 bytes 2024-09-16T23:19:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2177992704 bytes 2024-09-16T23:19:44.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_14 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:19:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:19:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-16T23:19:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_14': 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13188c9c7934 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13188c9c7934 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:44 2024 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:44 2024 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:44 2024 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ca2b9bfb-9340-437f-9dcd-274b942d579d 2024-09-16T23:19:45.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-16T23:19:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:19:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:19:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 62425088 bytes 2024-09-16T23:19:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 81683456 bytes 2024-09-16T23:19:46.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 131504128 bytes 2024-09-16T23:19:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 176312320 bytes 2024-09-16T23:19:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 221704192 bytes 2024-09-16T23:19:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 262494208 bytes 2024-09-16T23:19:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 296760320 bytes 2024-09-16T23:19:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 361692160 bytes 2024-09-16T23:19:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 439307264 bytes 2024-09-16T23:19:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 459890688 bytes 2024-09-16T23:19:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 554766336 bytes 2024-09-16T23:19:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 560319488 bytes 2024-09-16T23:19:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 582624256 bytes 2024-09-16T23:19:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 613019648 bytes 2024-09-16T23:19:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 633829376 bytes 2024-09-16T23:19:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 655663104 bytes 2024-09-16T23:19:48.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 697867264 bytes 2024-09-16T23:19:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 730916864 bytes 2024-09-16T23:19:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 744881152 bytes 2024-09-16T23:19:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 794362880 bytes 2024-09-16T23:19:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 857019392 bytes 2024-09-16T23:19:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 873723904 bytes 2024-09-16T23:19:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 889906176 bytes 2024-09-16T23:19:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 906244096 bytes 2024-09-16T23:19:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 985741312 bytes 2024-09-16T23:19:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1002552320 bytes 2024-09-16T23:19:49.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1082670080 bytes 2024-09-16T23:19:49.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1090230272 bytes 2024-09-16T23:19:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1132940288 bytes 2024-09-16T23:19:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1133550592 bytes 2024-09-16T23:19:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1254900736 bytes 2024-09-16T23:19:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1293669376 bytes 2024-09-16T23:19:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1321549824 bytes 2024-09-16T23:19:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1355393024 bytes 2024-09-16T23:19:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1443186688 bytes 2024-09-16T23:19:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1455774720 bytes 2024-09-16T23:19:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1541769216 bytes 2024-09-16T23:19:50.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1590640640 bytes 2024-09-16T23:19:50.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1631509504 bytes 2024-09-16T23:19:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1682651136 bytes 2024-09-16T23:19:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1683254272 bytes 2024-09-16T23:19:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1746677760 bytes 2024-09-16T23:19:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1748184064 bytes 2024-09-16T23:19:51.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1791047680 bytes 2024-09-16T23:19:51.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1822871552 bytes 2024-09-16T23:19:51.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1866175488 bytes 2024-09-16T23:19:51.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1887286272 bytes 2024-09-16T23:19:51.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1933815808 bytes 2024-09-16T23:19:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2005619712 bytes 2024-09-16T23:19:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2084675584 bytes 2024-09-16T23:19:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2098511872 bytes 2024-09-16T23:19:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2114123776 bytes 2024-09-16T23:19:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2166971392 bytes 2024-09-16T23:19:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2211543040 bytes 2024-09-16T23:19:52.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2280482816 bytes 2024-09-16T23:19:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2301333504 bytes 2024-09-16T23:19:52.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2332563456 bytes 2024-09-16T23:19:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2363841536 bytes 2024-09-16T23:19:53.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2398028800 bytes 2024-09-16T23:19:53.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2429121536 bytes 2024-09-16T23:19:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2467666944 bytes 2024-09-16T23:19:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2527198208 bytes 2024-09-16T23:19:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2575337472 bytes 2024-09-16T23:19:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2581804032 bytes 2024-09-16T23:19:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:19:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_15 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:19:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2024-09-16T23:19:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:19:53.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:19:53.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-16T23:19:54.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:19:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-16T23:19:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_15': 2024-09-16T23:19:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:19:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13244338ad89 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13244338ad89 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:53 2024 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:53 2024 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:53 2024 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: d611b2eb-d352-4abb-977d-a94f77c573c9 2024-09-16T23:19:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:19:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:19:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:19:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 624640 bytes 2024-09-16T23:19:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 1244160 bytes 2024-09-16T23:19:55.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 48124928 bytes 2024-09-16T23:19:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 61428736 bytes 2024-09-16T23:19:55.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 105575424 bytes 2024-09-16T23:19:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 185376768 bytes 2024-09-16T23:19:55.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 236752896 bytes 2024-09-16T23:19:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 257329152 bytes 2024-09-16T23:19:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 345577472 bytes 2024-09-16T23:19:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 404076544 bytes 2024-09-16T23:19:56.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 445793280 bytes 2024-09-16T23:19:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 510715904 bytes 2024-09-16T23:19:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 527352832 bytes 2024-09-16T23:19:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 579245056 bytes 2024-09-16T23:19:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 626590720 bytes 2024-09-16T23:19:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 689039360 bytes 2024-09-16T23:19:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 722529280 bytes 2024-09-16T23:19:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 745389056 bytes 2024-09-16T23:19:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 810586112 bytes 2024-09-16T23:19:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 832021504 bytes 2024-09-16T23:19:57.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 849164288 bytes 2024-09-16T23:19:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 878158848 bytes 2024-09-16T23:19:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 893030400 bytes 2024-09-16T23:19:57.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 954229760 bytes 2024-09-16T23:19:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1064830976 bytes 2024-09-16T23:19:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1130707968 bytes 2024-09-16T23:19:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1137068032 bytes 2024-09-16T23:19:58.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1169119232 bytes 2024-09-16T23:19:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1169720320 bytes 2024-09-16T23:19:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1197117440 bytes 2024-09-16T23:19:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1225657344 bytes 2024-09-16T23:19:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1278656512 bytes 2024-09-16T23:19:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1322418176 bytes 2024-09-16T23:19:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1328661504 bytes 2024-09-16T23:19:59.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1352480768 bytes 2024-09-16T23:19:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1438349312 bytes 2024-09-16T23:19:59.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1467720704 bytes 2024-09-16T23:19:59.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1489393664 bytes 2024-09-16T23:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1531019264 bytes 2024-09-16T23:19:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1576725504 bytes 2024-09-16T23:19:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1628407808 bytes 2024-09-16T23:19:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1630124032 bytes 2024-09-16T23:20:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1745012736 bytes 2024-09-16T23:20:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1784215552 bytes 2024-09-16T23:20:00.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1816190976 bytes 2024-09-16T23:20:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1888535552 bytes 2024-09-16T23:20:00.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1927791616 bytes 2024-09-16T23:20:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1989738496 bytes 2024-09-16T23:20:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2028120064 bytes 2024-09-16T23:20:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2097602560 bytes 2024-09-16T23:20:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2100186112 bytes 2024-09-16T23:20:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2136669184 bytes 2024-09-16T23:20:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2161701888 bytes 2024-09-16T23:20:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2165633024 bytes 2024-09-16T23:20:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2212086784 bytes 2024-09-16T23:20:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2212899840 bytes 2024-09-16T23:20:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2232459264 bytes 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_16 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:01.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:01.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-16T23:20:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:02.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-16T23:20:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_16': 2024-09-16T23:20:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13303f53db94 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13303f53db94 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:01 2024 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:01 2024 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:01 2024 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 161cd388-c85e-48c4-8c8e-84aaca587105 2024-09-16T23:20:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-16T23:20:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:20:02.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:20:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:20:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 32890880 bytes 2024-09-16T23:20:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 33504256 bytes 2024-09-16T23:20:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 98340864 bytes 2024-09-16T23:20:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 130711552 bytes 2024-09-16T23:20:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 141395968 bytes 2024-09-16T23:20:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 165287936 bytes 2024-09-16T23:20:03.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 197724160 bytes 2024-09-16T23:20:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 216357888 bytes 2024-09-16T23:20:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 249202688 bytes 2024-09-16T23:20:03.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 322554880 bytes 2024-09-16T23:20:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 332919808 bytes 2024-09-16T23:20:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 359924736 bytes 2024-09-16T23:20:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 374823936 bytes 2024-09-16T23:20:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 436086784 bytes 2024-09-16T23:20:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 446437376 bytes 2024-09-16T23:20:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 467856384 bytes 2024-09-16T23:20:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 486396928 bytes 2024-09-16T23:20:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 491188224 bytes 2024-09-16T23:20:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 535497728 bytes 2024-09-16T23:20:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 589383680 bytes 2024-09-16T23:20:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 668641280 bytes 2024-09-16T23:20:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 686389248 bytes 2024-09-16T23:20:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 706452480 bytes 2024-09-16T23:20:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 748114944 bytes 2024-09-16T23:20:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 771504128 bytes 2024-09-16T23:20:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 863260672 bytes 2024-09-16T23:20:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 863854592 bytes 2024-09-16T23:20:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 880054272 bytes 2024-09-16T23:20:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 903247872 bytes 2024-09-16T23:20:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 969715712 bytes 2024-09-16T23:20:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 994709504 bytes 2024-09-16T23:20:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1064267776 bytes 2024-09-16T23:20:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1116580864 bytes 2024-09-16T23:20:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1146567680 bytes 2024-09-16T23:20:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1198360576 bytes 2024-09-16T23:20:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1320328192 bytes 2024-09-16T23:20:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1334219776 bytes 2024-09-16T23:20:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1368866816 bytes 2024-09-16T23:20:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1387929600 bytes 2024-09-16T23:20:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1420697600 bytes 2024-09-16T23:20:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1467362304 bytes 2024-09-16T23:20:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1507166208 bytes 2024-09-16T23:20:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1596339200 bytes 2024-09-16T23:20:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1679071232 bytes 2024-09-16T23:20:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1705910272 bytes 2024-09-16T23:20:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1742735360 bytes 2024-09-16T23:20:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1778526208 bytes 2024-09-16T23:20:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1788657664 bytes 2024-09-16T23:20:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1855967232 bytes 2024-09-16T23:20:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1937900544 bytes 2024-09-16T23:20:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1962919936 bytes 2024-09-16T23:20:08.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2026438656 bytes 2024-09-16T23:20:08.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2061536256 bytes 2024-09-16T23:20:09.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2070820864 bytes 2024-09-16T23:20:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2099588096 bytes 2024-09-16T23:20:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2201616384 bytes 2024-09-16T23:20:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2268502016 bytes 2024-09-16T23:20:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2305670144 bytes 2024-09-16T23:20:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2306272256 bytes 2024-09-16T23:20:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2368371712 bytes 2024-09-16T23:20:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2445670400 bytes 2024-09-16T23:20:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2484471808 bytes 2024-09-16T23:20:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2498321408 bytes 2024-09-16T23:20:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2525303808 bytes 2024-09-16T23:20:10.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2588329984 bytes 2024-09-16T23:20:10.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2595018752 bytes 2024-09-16T23:20:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2661018624 bytes 2024-09-16T23:20:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2734590976 bytes 2024-09-16T23:20:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2741141504 bytes 2024-09-16T23:20:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2768396288 bytes 2024-09-16T23:20:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2777499648 bytes 2024-09-16T23:20:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2778103808 bytes 2024-09-16T23:20:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2795000832 bytes 2024-09-16T23:20:11.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2824961024 bytes 2024-09-16T23:20:11.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2887035904 bytes 2024-09-16T23:20:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2928185344 bytes 2024-09-16T23:20:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 2992571392 bytes 2024-09-16T23:20:11.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3003993088 bytes 2024-09-16T23:20:12.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3004592128 bytes 2024-09-16T23:20:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3056052224 bytes 2024-09-16T23:20:12.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3113068544 bytes 2024-09-16T23:20:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3181253632 bytes 2024-09-16T23:20:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3229453312 bytes 2024-09-16T23:20:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3268281344 bytes 2024-09-16T23:20:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_17 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2024-09-16T23:20:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-16T23:20:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:13.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-16T23:20:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_17': 2024-09-16T23:20:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 133c6db29813 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.133c6db29813 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:12 2024 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:12 2024 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:12 2024 2024-09-16T23:20:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 77e8cc56-9128-4126-9b3a-02b903130874 2024-09-16T23:20:13.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:20:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:20:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:13.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:20:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 19329024 bytes 2024-09-16T23:20:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 116075520 bytes 2024-09-16T23:20:14.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 146328576 bytes 2024-09-16T23:20:14.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 185256960 bytes 2024-09-16T23:20:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 204803072 bytes 2024-09-16T23:20:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 244404224 bytes 2024-09-16T23:20:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 316850176 bytes 2024-09-16T23:20:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 374777856 bytes 2024-09-16T23:20:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 398564352 bytes 2024-09-16T23:20:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 463494144 bytes 2024-09-16T23:20:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 474133504 bytes 2024-09-16T23:20:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 501073920 bytes 2024-09-16T23:20:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 568579072 bytes 2024-09-16T23:20:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 651101184 bytes 2024-09-16T23:20:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 677598208 bytes 2024-09-16T23:20:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 723700736 bytes 2024-09-16T23:20:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 724319232 bytes 2024-09-16T23:20:15.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 753200128 bytes 2024-09-16T23:20:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 803889152 bytes 2024-09-16T23:20:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 837318656 bytes 2024-09-16T23:20:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 891251712 bytes 2024-09-16T23:20:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 951533568 bytes 2024-09-16T23:20:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 952916992 bytes 2024-09-16T23:20:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 993902592 bytes 2024-09-16T23:20:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1008038912 bytes 2024-09-16T23:20:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1081598976 bytes 2024-09-16T23:20:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1095902208 bytes 2024-09-16T23:20:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1120946176 bytes 2024-09-16T23:20:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1167175680 bytes 2024-09-16T23:20:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1191471104 bytes 2024-09-16T23:20:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1258000384 bytes 2024-09-16T23:20:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1305447424 bytes 2024-09-16T23:20:17.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1363054592 bytes 2024-09-16T23:20:17.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1372849152 bytes 2024-09-16T23:20:17.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1393016832 bytes 2024-09-16T23:20:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1396856832 bytes 2024-09-16T23:20:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1449552896 bytes 2024-09-16T23:20:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1491243008 bytes 2024-09-16T23:20:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1508502528 bytes 2024-09-16T23:20:18.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1530456064 bytes 2024-09-16T23:20:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1563951104 bytes 2024-09-16T23:20:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1610273792 bytes 2024-09-16T23:20:18.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1630869504 bytes 2024-09-16T23:20:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1631489024 bytes 2024-09-16T23:20:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1698940928 bytes 2024-09-16T23:20:19.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1760166912 bytes 2024-09-16T23:20:19.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1807728640 bytes 2024-09-16T23:20:19.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1835113472 bytes 2024-09-16T23:20:19.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1877855232 bytes 2024-09-16T23:20:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1929762816 bytes 2024-09-16T23:20:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2019336192 bytes 2024-09-16T23:20:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2110258176 bytes 2024-09-16T23:20:20.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2164443136 bytes 2024-09-16T23:20:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2198619136 bytes 2024-09-16T23:20:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2217924608 bytes 2024-09-16T23:20:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2290229248 bytes 2024-09-16T23:20:20.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2338738176 bytes 2024-09-16T23:20:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2339308544 bytes 2024-09-16T23:20:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2380163072 bytes 2024-09-16T23:20:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2443560960 bytes 2024-09-16T23:20:21.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2457318400 bytes 2024-09-16T23:20:21.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2481205248 bytes 2024-09-16T23:20:21.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2509335552 bytes 2024-09-16T23:20:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2584886272 bytes 2024-09-16T23:20:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2593801216 bytes 2024-09-16T23:20:21.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2696802304 bytes 2024-09-16T23:20:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2706579456 bytes 2024-09-16T23:20:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2786513920 bytes 2024-09-16T23:20:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2808233984 bytes 2024-09-16T23:20:22.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2832847872 bytes 2024-09-16T23:20:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2856611840 bytes 2024-09-16T23:20:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2878230528 bytes 2024-09-16T23:20:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2952324096 bytes 2024-09-16T23:20:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2969253888 bytes 2024-09-16T23:20:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3052322816 bytes 2024-09-16T23:20:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3083278336 bytes 2024-09-16T23:20:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3128762368 bytes 2024-09-16T23:20:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3182782464 bytes 2024-09-16T23:20:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3224471552 bytes 2024-09-16T23:20:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3276261376 bytes 2024-09-16T23:20:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3324554240 bytes 2024-09-16T23:20:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3400123392 bytes 2024-09-16T23:20:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3463317504 bytes 2024-09-16T23:20:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3517747200 bytes 2024-09-16T23:20:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3546989568 bytes 2024-09-16T23:20:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3598324736 bytes 2024-09-16T23:20:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3629408256 bytes 2024-09-16T23:20:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3663519744 bytes 2024-09-16T23:20:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3707423744 bytes 2024-09-16T23:20:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3794881536 bytes 2024-09-16T23:20:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3809283072 bytes 2024-09-16T23:20:25.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3887584256 bytes 2024-09-16T23:20:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3962071040 bytes 2024-09-16T23:20:25.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3994644480 bytes 2024-09-16T23:20:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4006902784 bytes 2024-09-16T23:20:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4035270656 bytes 2024-09-16T23:20:25.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4076418048 bytes 2024-09-16T23:20:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4126400512 bytes 2024-09-16T23:20:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4157021184 bytes 2024-09-16T23:20:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4183985152 bytes 2024-09-16T23:20:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4244210688 bytes 2024-09-16T23:20:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4258776064 bytes 2024-09-16T23:20:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4320652288 bytes 2024-09-16T23:20:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4360360960 bytes 2024-09-16T23:20:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4413497344 bytes 2024-09-16T23:20:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4446867456 bytes 2024-09-16T23:20:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4468017152 bytes 2024-09-16T23:20:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4559665152 bytes 2024-09-16T23:20:27.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4599552000 bytes 2024-09-16T23:20:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4630449152 bytes 2024-09-16T23:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4671356928 bytes 2024-09-16T23:20:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4712636416 bytes 2024-09-16T23:20:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4761707520 bytes 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_18 2024-09-16T23:20:27.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:27.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-16T23:20:28.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:28.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-16T23:20:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_18': 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13488ea4bd99 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13488ea4bd99 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:27 2024 2024-09-16T23:20:28.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:27 2024 2024-09-16T23:20:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:27 2024 2024-09-16T23:20:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 58af7715-4fc6-402e-aa27-d73e969a9d7a 2024-09-16T23:20:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-16T23:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:20:28.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=9 2024-09-16T23:20:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:20:28.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 2196480 bytes 2024-09-16T23:20:29.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 23682048 bytes 2024-09-16T23:20:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 60537856 bytes 2024-09-16T23:20:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 134504448 bytes 2024-09-16T23:20:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 148031488 bytes 2024-09-16T23:20:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 217006080 bytes 2024-09-16T23:20:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 229962752 bytes 2024-09-16T23:20:29.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 260359168 bytes 2024-09-16T23:20:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 290749440 bytes 2024-09-16T23:20:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 329777152 bytes 2024-09-16T23:20:30.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 346934272 bytes 2024-09-16T23:20:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 352867328 bytes 2024-09-16T23:20:30.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 369931264 bytes 2024-09-16T23:20:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 409997312 bytes 2024-09-16T23:20:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 421612544 bytes 2024-09-16T23:20:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 516520960 bytes 2024-09-16T23:20:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 548736000 bytes 2024-09-16T23:20:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 572913664 bytes 2024-09-16T23:20:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 597463040 bytes 2024-09-16T23:20:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 615209984 bytes 2024-09-16T23:20:31.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 697789440 bytes 2024-09-16T23:20:31.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 867947520 bytes 2024-09-16T23:20:31.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 883529728 bytes 2024-09-16T23:20:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 927726592 bytes 2024-09-16T23:20:31.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 985863168 bytes 2024-09-16T23:20:31.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1027378176 bytes 2024-09-16T23:20:32.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1071303680 bytes 2024-09-16T23:20:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1160366080 bytes 2024-09-16T23:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1180634112 bytes 2024-09-16T23:20:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1226904576 bytes 2024-09-16T23:20:32.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1244069888 bytes 2024-09-16T23:20:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1269222400 bytes 2024-09-16T23:20:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1279416320 bytes 2024-09-16T23:20:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1351183360 bytes 2024-09-16T23:20:33.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1385760768 bytes 2024-09-16T23:20:33.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1490841600 bytes 2024-09-16T23:20:33.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1501128704 bytes 2024-09-16T23:20:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1527589888 bytes 2024-09-16T23:20:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1559300096 bytes 2024-09-16T23:20:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1586727936 bytes 2024-09-16T23:20:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1642961920 bytes 2024-09-16T23:20:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1709129728 bytes 2024-09-16T23:20:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1772030976 bytes 2024-09-16T23:20:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1794476032 bytes 2024-09-16T23:20:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1834212352 bytes 2024-09-16T23:20:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1862177792 bytes 2024-09-16T23:20:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1877919744 bytes 2024-09-16T23:20:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1936416768 bytes 2024-09-16T23:20:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1986018304 bytes 2024-09-16T23:20:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2007518208 bytes 2024-09-16T23:20:34.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2026804224 bytes 2024-09-16T23:20:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2060048384 bytes 2024-09-16T23:20:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2088038400 bytes 2024-09-16T23:20:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2143057920 bytes 2024-09-16T23:20:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2143664128 bytes 2024-09-16T23:20:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2163147776 bytes 2024-09-16T23:20:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2196198400 bytes 2024-09-16T23:20:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2262309888 bytes 2024-09-16T23:20:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2304530432 bytes 2024-09-16T23:20:36.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2359515136 bytes 2024-09-16T23:20:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2366854144 bytes 2024-09-16T23:20:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2436429824 bytes 2024-09-16T23:20:36.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2555654144 bytes 2024-09-16T23:20:36.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2562847744 bytes 2024-09-16T23:20:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2626203648 bytes 2024-09-16T23:20:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2670371840 bytes 2024-09-16T23:20:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2711137280 bytes 2024-09-16T23:20:37.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2785933312 bytes 2024-09-16T23:20:37.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2816504832 bytes 2024-09-16T23:20:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2846801920 bytes 2024-09-16T23:20:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2870078464 bytes 2024-09-16T23:20:37.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2903150592 bytes 2024-09-16T23:20:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2948477952 bytes 2024-09-16T23:20:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2970059776 bytes 2024-09-16T23:20:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_19 2024-09-16T23:20:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2024-09-16T23:20:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:37.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-16T23:20:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:20:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-16T23:20:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_19': 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13542846b480 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13542846b480 2024-09-16T23:20:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:37 2024 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:37 2024 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:37 2024 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 76b07c29-6ac0-440f-9c18-26ac80777711 2024-09-16T23:20:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-16T23:20:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:20:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=5 2024-09-16T23:20:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:20:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 5387264 bytes 2024-09-16T23:20:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 43882496 bytes 2024-09-16T23:20:39.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 94746624 bytes 2024-09-16T23:20:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 147130368 bytes 2024-09-16T23:20:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 201893888 bytes 2024-09-16T23:20:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 239880192 bytes 2024-09-16T23:20:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 273166336 bytes 2024-09-16T23:20:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 300254208 bytes 2024-09-16T23:20:39.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 304710656 bytes 2024-09-16T23:20:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 305352704 bytes 2024-09-16T23:20:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 390479872 bytes 2024-09-16T23:20:40.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 406488064 bytes 2024-09-16T23:20:40.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 432156672 bytes 2024-09-16T23:20:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 489411584 bytes 2024-09-16T23:20:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 584607744 bytes 2024-09-16T23:20:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 611707904 bytes 2024-09-16T23:20:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 651898880 bytes 2024-09-16T23:20:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 661061632 bytes 2024-09-16T23:20:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 695844864 bytes 2024-09-16T23:20:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 785403904 bytes 2024-09-16T23:20:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 892712960 bytes 2024-09-16T23:20:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 933905408 bytes 2024-09-16T23:20:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 976405504 bytes 2024-09-16T23:20:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1001631744 bytes 2024-09-16T23:20:41.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1031229440 bytes 2024-09-16T23:20:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1053193216 bytes 2024-09-16T23:20:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1132646400 bytes 2024-09-16T23:20:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1133256704 bytes 2024-09-16T23:20:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1159213056 bytes 2024-09-16T23:20:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1221579776 bytes 2024-09-16T23:20:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1300709376 bytes 2024-09-16T23:20:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1334854656 bytes 2024-09-16T23:20:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1381794816 bytes 2024-09-16T23:20:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1427852288 bytes 2024-09-16T23:20:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1521026048 bytes 2024-09-16T23:20:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1565574144 bytes 2024-09-16T23:20:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1626595328 bytes 2024-09-16T23:20:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1639141376 bytes 2024-09-16T23:20:43.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1768785920 bytes 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_20 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-16T23:20:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-16T23:20:43.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:43.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:43.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:20:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-16T23:20:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-16T23:20:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_20': 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13609c4cd0b7 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13609c4cd0b7 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:43 2024 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:43 2024 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:43 2024 2024-09-16T23:20:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3e27fd33-23b8-4a4b-a8ee-9baf2df8b8f0 2024-09-16T23:20:44.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:20:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:20:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:20:45.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 51845120 bytes 2024-09-16T23:20:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 104187904 bytes 2024-09-16T23:20:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 126816256 bytes 2024-09-16T23:20:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 232976384 bytes 2024-09-16T23:20:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 258201600 bytes 2024-09-16T23:20:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 346498048 bytes 2024-09-16T23:20:45.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 408012800 bytes 2024-09-16T23:20:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 465912832 bytes 2024-09-16T23:20:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 496609280 bytes 2024-09-16T23:20:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 560381952 bytes 2024-09-16T23:20:46.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 592309248 bytes 2024-09-16T23:20:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 701875200 bytes 2024-09-16T23:20:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 726375424 bytes 2024-09-16T23:20:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 807270400 bytes 2024-09-16T23:20:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 896834560 bytes 2024-09-16T23:20:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 920297472 bytes 2024-09-16T23:20:47.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 933596160 bytes 2024-09-16T23:20:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 993992704 bytes 2024-09-16T23:20:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 1054959616 bytes 2024-09-16T23:20:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1133652992 bytes 2024-09-16T23:20:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1161810944 bytes 2024-09-16T23:20:47.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1218072576 bytes 2024-09-16T23:20:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1243118592 bytes 2024-09-16T23:20:48.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1263443968 bytes 2024-09-16T23:20:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1320083456 bytes 2024-09-16T23:20:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1401972736 bytes 2024-09-16T23:20:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1444295680 bytes 2024-09-16T23:20:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1481620480 bytes 2024-09-16T23:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1496759296 bytes 2024-09-16T23:20:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1528125440 bytes 2024-09-16T23:20:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1561196544 bytes 2024-09-16T23:20:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1642302464 bytes 2024-09-16T23:20:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1692706816 bytes 2024-09-16T23:20:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1715962880 bytes 2024-09-16T23:20:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1794737152 bytes 2024-09-16T23:20:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1932535808 bytes 2024-09-16T23:20:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1960629248 bytes 2024-09-16T23:20:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 2002118656 bytes 2024-09-16T23:20:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 2017605632 bytes 2024-09-16T23:20:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 2051313664 bytes 2024-09-16T23:20:50.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 2079771648 bytes 2024-09-16T23:20:50.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2108408832 bytes 2024-09-16T23:20:50.355 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.946741781717655) > weight (0.25) 2024-09-16T23:20:50.356 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T23:20:50.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T23:20:50.357 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:20:50.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:20:50.359+0000 7fabb4873640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 36971) UID: 1000 2024-09-16T23:20:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2155012096 bytes 2024-09-16T23:20:50.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2179049472 bytes 2024-09-16T23:20:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2249409536 bytes 2024-09-16T23:20:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2304785408 bytes 2024-09-16T23:20:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2385095680 bytes 2024-09-16T23:20:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2416824320 bytes 2024-09-16T23:20:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2451590144 bytes 2024-09-16T23:20:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2484804608 bytes 2024-09-16T23:20:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2515024896 bytes 2024-09-16T23:20:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2535308288 bytes 2024-09-16T23:20:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2589971456 bytes 2024-09-16T23:20:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2627340288 bytes 2024-09-16T23:20:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2663434240 bytes 2024-09-16T23:20:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2681407488 bytes 2024-09-16T23:20:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2738588672 bytes 2024-09-16T23:20:52.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2760676352 bytes 2024-09-16T23:20:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2780996608 bytes 2024-09-16T23:20:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2847309824 bytes 2024-09-16T23:20:52.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2887957504 bytes 2024-09-16T23:20:52.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2918222848 bytes 2024-09-16T23:20:52.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2943166464 bytes 2024-09-16T23:20:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2970494976 bytes 2024-09-16T23:20:53.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2987490304 bytes 2024-09-16T23:20:53.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3023769600 bytes 2024-09-16T23:20:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3062313984 bytes 2024-09-16T23:20:53.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3072556032 bytes 2024-09-16T23:20:53.447 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T23:20:53.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T23:20:53.447 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T23:20:53.448 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:20:53.450+0000 7f1139c50640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42228) UID: 1000 2024-09-16T23:20:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3076367360 bytes 2024-09-16T23:20:53.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3076970496 bytes 2024-09-16T23:20:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3095779328 bytes 2024-09-16T23:20:53.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3202368512 bytes 2024-09-16T23:20:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3237199872 bytes 2024-09-16T23:20:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3256628224 bytes 2024-09-16T23:20:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3306841088 bytes 2024-09-16T23:20:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3341456384 bytes 2024-09-16T23:20:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3395512320 bytes 2024-09-16T23:20:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3460197376 bytes 2024-09-16T23:20:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3534431232 bytes 2024-09-16T23:20:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3622574080 bytes 2024-09-16T23:20:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3663448064 bytes 2024-09-16T23:20:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3707556864 bytes 2024-09-16T23:20:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3723555840 bytes 2024-09-16T23:20:55.358 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T23:20:55.358 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:20:55.358 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T23:20:55.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T23:20:55.359 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T23:20:55.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T23:20:55.364 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2024-09-16T23:20:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3758495744 bytes 2024-09-16T23:20:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3828129792 bytes 2024-09-16T23:20:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3859166208 bytes 2024-09-16T23:20:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3898431488 bytes 2024-09-16T23:20:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3899106304 bytes 2024-09-16T23:20:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3913881600 bytes 2024-09-16T23:20:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3914472448 bytes 2024-09-16T23:20:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3969459200 bytes 2024-09-16T23:20:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3985231872 bytes 2024-09-16T23:20:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4086162432 bytes 2024-09-16T23:20:56.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4086771712 bytes 2024-09-16T23:20:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4120510464 bytes 2024-09-16T23:20:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4121351168 bytes 2024-09-16T23:20:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4128011264 bytes 2024-09-16T23:20:57.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4195266560 bytes 2024-09-16T23:20:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4275075072 bytes 2024-09-16T23:20:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4330775552 bytes 2024-09-16T23:20:57.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4430042112 bytes 2024-09-16T23:20:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4447350784 bytes 2024-09-16T23:20:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4485197824 bytes 2024-09-16T23:20:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4498836480 bytes 2024-09-16T23:20:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4525271040 bytes 2024-09-16T23:20:58.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4568867840 bytes 2024-09-16T23:20:58.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4591432704 bytes 2024-09-16T23:20:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4594569216 bytes 2024-09-16T23:20:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4595450880 bytes 2024-09-16T23:20:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4632742912 bytes 2024-09-16T23:20:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4669447168 bytes 2024-09-16T23:20:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4705817600 bytes 2024-09-16T23:20:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4725770240 bytes 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_21 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:20:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:20:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:20:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:20:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:20:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2024-09-16T23:20:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:20:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:20:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-16T23:20:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:20:59.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_21': 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 137515347aeb 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.137515347aeb 2024-09-16T23:20:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:58 2024 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:58 2024 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:58 2024 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: eecd37c8-2507-4ea3-8c85-72ea35b897c6 2024-09-16T23:20:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:20:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:20:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-16T23:20:59.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:20:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:20:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:20:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:20:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:20:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:20:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 52231168 bytes 2024-09-16T23:21:00.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 97517568 bytes 2024-09-16T23:21:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 125062144 bytes 2024-09-16T23:21:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 140265472 bytes 2024-09-16T23:21:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 233843712 bytes 2024-09-16T23:21:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 287078400 bytes 2024-09-16T23:21:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 319977472 bytes 2024-09-16T23:21:00.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 399394816 bytes 2024-09-16T23:21:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 477745152 bytes 2024-09-16T23:21:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 496079872 bytes 2024-09-16T23:21:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 534682624 bytes 2024-09-16T23:21:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 549381120 bytes 2024-09-16T23:21:01.448 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T23:21:01.448 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:21:01.449 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T23:21:01.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T23:21:01.449 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T23:21:01.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T23:21:01.452 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-16T23:21:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 579184640 bytes 2024-09-16T23:21:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 630298624 bytes 2024-09-16T23:21:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 659336192 bytes 2024-09-16T23:21:01.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 724833280 bytes 2024-09-16T23:21:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 778564608 bytes 2024-09-16T23:21:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 828243968 bytes 2024-09-16T23:21:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 873808896 bytes 2024-09-16T23:21:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 911019008 bytes 2024-09-16T23:21:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 911650816 bytes 2024-09-16T23:21:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 912285696 bytes 2024-09-16T23:21:02.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 934481920 bytes 2024-09-16T23:21:02.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 949239808 bytes 2024-09-16T23:21:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 994774016 bytes 2024-09-16T23:21:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1007582208 bytes 2024-09-16T23:21:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1046569984 bytes 2024-09-16T23:21:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1059760128 bytes 2024-09-16T23:21:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1197786112 bytes 2024-09-16T23:21:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1229233152 bytes 2024-09-16T23:21:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1314494464 bytes 2024-09-16T23:21:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1352612864 bytes 2024-09-16T23:21:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1373168640 bytes 2024-09-16T23:21:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1394297856 bytes 2024-09-16T23:21:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1431522304 bytes 2024-09-16T23:21:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1481779200 bytes 2024-09-16T23:21:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1585617920 bytes 2024-09-16T23:21:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1674404864 bytes 2024-09-16T23:21:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1715624960 bytes 2024-09-16T23:21:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1736361984 bytes 2024-09-16T23:21:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1806779392 bytes 2024-09-16T23:21:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1854345216 bytes 2024-09-16T23:21:05.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1918771200 bytes 2024-09-16T23:21:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2013210624 bytes 2024-09-16T23:21:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2039556096 bytes 2024-09-16T23:21:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2117494784 bytes 2024-09-16T23:21:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2168554496 bytes 2024-09-16T23:21:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2221025280 bytes 2024-09-16T23:21:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2245309440 bytes 2024-09-16T23:21:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2270315520 bytes 2024-09-16T23:21:06.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2270912512 bytes 2024-09-16T23:21:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2372646912 bytes 2024-09-16T23:21:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2394976256 bytes 2024-09-16T23:21:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2475405312 bytes 2024-09-16T23:21:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2548115456 bytes 2024-09-16T23:21:07.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2594464768 bytes 2024-09-16T23:21:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2672180224 bytes 2024-09-16T23:21:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2758588416 bytes 2024-09-16T23:21:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2821344256 bytes 2024-09-16T23:21:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2886502400 bytes 2024-09-16T23:21:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2951592960 bytes 2024-09-16T23:21:07.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2990387200 bytes 2024-09-16T23:21:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2996078592 bytes 2024-09-16T23:21:08.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3048047616 bytes 2024-09-16T23:21:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3141482496 bytes 2024-09-16T23:21:08.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3185283072 bytes 2024-09-16T23:21:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3239254016 bytes 2024-09-16T23:21:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3258042368 bytes 2024-09-16T23:21:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3325053952 bytes 2024-09-16T23:21:08.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3391515648 bytes 2024-09-16T23:21:09.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3441594368 bytes 2024-09-16T23:21:09.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3462027264 bytes 2024-09-16T23:21:09.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3545584640 bytes 2024-09-16T23:21:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3564403712 bytes 2024-09-16T23:21:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3594074112 bytes 2024-09-16T23:21:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3628177408 bytes 2024-09-16T23:21:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3661139968 bytes 2024-09-16T23:21:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3713807360 bytes 2024-09-16T23:21:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3749980160 bytes 2024-09-16T23:21:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3810407424 bytes 2024-09-16T23:21:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3830785024 bytes 2024-09-16T23:21:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3860205568 bytes 2024-09-16T23:21:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3873031168 bytes 2024-09-16T23:21:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3878801408 bytes 2024-09-16T23:21:10.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3957852160 bytes 2024-09-16T23:21:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 4016609280 bytes 2024-09-16T23:21:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 4017213440 bytes 2024-09-16T23:21:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 4076488704 bytes 2024-09-16T23:21:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 4125390848 bytes 2024-09-16T23:21:11.355 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4148157440 bytes 2024-09-16T23:21:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4149166080 bytes 2024-09-16T23:21:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4151835648 bytes 2024-09-16T23:21:11.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4230312960 bytes 2024-09-16T23:21:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4272156672 bytes 2024-09-16T23:21:12.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4359323648 bytes 2024-09-16T23:21:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4399034368 bytes 2024-09-16T23:21:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4425985024 bytes 2024-09-16T23:21:12.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4488705024 bytes 2024-09-16T23:21:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4544545792 bytes 2024-09-16T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:21:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_22 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:21:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2024-09-16T23:21:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:21:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:21:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-16T23:21:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:21:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-16T23:21:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:21:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-16T23:21:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_22': 2024-09-16T23:21:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:21:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:21:13.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1385d7584f52 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1385d7584f52 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:13 2024 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:13 2024 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:13 2024 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: fb737a24-defd-44de-8405-d2f600fcfa9d 2024-09-16T23:21:13.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:21:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:21:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-16T23:21:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:21:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:21:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:21:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:21:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 62937088 bytes 2024-09-16T23:21:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 143794176 bytes 2024-09-16T23:21:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 159063040 bytes 2024-09-16T23:21:14.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 182881280 bytes 2024-09-16T23:21:14.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 251183104 bytes 2024-09-16T23:21:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 284832768 bytes 2024-09-16T23:21:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 339857408 bytes 2024-09-16T23:21:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 370270208 bytes 2024-09-16T23:21:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 370874368 bytes 2024-09-16T23:21:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 388026368 bytes 2024-09-16T23:21:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 397552640 bytes 2024-09-16T23:21:15.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 481959936 bytes 2024-09-16T23:21:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 498998272 bytes 2024-09-16T23:21:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 592495616 bytes 2024-09-16T23:21:15.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 637937664 bytes 2024-09-16T23:21:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 645673984 bytes 2024-09-16T23:21:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 709278720 bytes 2024-09-16T23:21:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 729955328 bytes 2024-09-16T23:21:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 754316288 bytes 2024-09-16T23:21:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 805865472 bytes 2024-09-16T23:21:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 858473472 bytes 2024-09-16T23:21:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 881187840 bytes 2024-09-16T23:21:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 889676800 bytes 2024-09-16T23:21:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 970534912 bytes 2024-09-16T23:21:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1009611776 bytes 2024-09-16T23:21:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1057287168 bytes 2024-09-16T23:21:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1122441216 bytes 2024-09-16T23:21:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1163478016 bytes 2024-09-16T23:21:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1175767040 bytes 2024-09-16T23:21:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1177201664 bytes 2024-09-16T23:21:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1214961664 bytes 2024-09-16T23:21:17.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1294436352 bytes 2024-09-16T23:21:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1400157184 bytes 2024-09-16T23:21:18.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1414673408 bytes 2024-09-16T23:21:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1440976896 bytes 2024-09-16T23:21:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1509971968 bytes 2024-09-16T23:21:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1528634368 bytes 2024-09-16T23:21:18.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1580922880 bytes 2024-09-16T23:21:18.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1624072192 bytes 2024-09-16T23:21:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1734253568 bytes 2024-09-16T23:21:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1768451072 bytes 2024-09-16T23:21:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1805721600 bytes 2024-09-16T23:21:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1870522368 bytes 2024-09-16T23:21:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1918502912 bytes 2024-09-16T23:21:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1925011456 bytes 2024-09-16T23:21:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1955303424 bytes 2024-09-16T23:21:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1989724160 bytes 2024-09-16T23:21:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2058056704 bytes 2024-09-16T23:21:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2095194112 bytes 2024-09-16T23:21:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2124739584 bytes 2024-09-16T23:21:20.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2191479808 bytes 2024-09-16T23:21:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2207763456 bytes 2024-09-16T23:21:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2252515328 bytes 2024-09-16T23:21:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2268115968 bytes 2024-09-16T23:21:20.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2342334464 bytes 2024-09-16T23:21:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2363515904 bytes 2024-09-16T23:21:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2387346432 bytes 2024-09-16T23:21:21.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2459594752 bytes 2024-09-16T23:21:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2460211200 bytes 2024-09-16T23:21:21.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2516002816 bytes 2024-09-16T23:21:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2553299968 bytes 2024-09-16T23:21:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2652140544 bytes 2024-09-16T23:21:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2663983104 bytes 2024-09-16T23:21:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2724638720 bytes 2024-09-16T23:21:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2738666496 bytes 2024-09-16T23:21:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2774196224 bytes 2024-09-16T23:21:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2774808576 bytes 2024-09-16T23:21:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2815808512 bytes 2024-09-16T23:21:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2836861952 bytes 2024-09-16T23:21:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2841157632 bytes 2024-09-16T23:21:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2861089792 bytes 2024-09-16T23:21:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2930840576 bytes 2024-09-16T23:21:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2976120832 bytes 2024-09-16T23:21:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3066900480 bytes 2024-09-16T23:21:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3089070080 bytes 2024-09-16T23:21:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3104811008 bytes 2024-09-16T23:21:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3170396160 bytes 2024-09-16T23:21:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3231476736 bytes 2024-09-16T23:21:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3243689984 bytes 2024-09-16T23:21:23.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3262854144 bytes 2024-09-16T23:21:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3292497920 bytes 2024-09-16T23:21:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3293123584 bytes 2024-09-16T23:21:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3362238464 bytes 2024-09-16T23:21:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3407449088 bytes 2024-09-16T23:21:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3420778496 bytes 2024-09-16T23:21:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3458104320 bytes 2024-09-16T23:21:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3486961664 bytes 2024-09-16T23:21:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3564969984 bytes 2024-09-16T23:21:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3565585408 bytes 2024-09-16T23:21:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3608548352 bytes 2024-09-16T23:21:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3637991424 bytes 2024-09-16T23:21:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3669985280 bytes 2024-09-16T23:21:25.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3711736832 bytes 2024-09-16T23:21:25.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3724913664 bytes 2024-09-16T23:21:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3735616512 bytes 2024-09-16T23:21:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3756688384 bytes 2024-09-16T23:21:25.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 3832588288 bytes 2024-09-16T23:21:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 3881650176 bytes 2024-09-16T23:21:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_23 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:21:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:21:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:25.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-16T23:21:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:21:26.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-16T23:21:26.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_23': 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1391a924cb2 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1391a924cb2 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:25 2024 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:25 2024 2024-09-16T23:21:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:25 2024 2024-09-16T23:21:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:21:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:21:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: e1d61309-9267-41b0-b201-a6011fded361 2024-09-16T23:21:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:21:26.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:21:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:21:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:21:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:27.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 25043968 bytes 2024-09-16T23:21:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 54461440 bytes 2024-09-16T23:21:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 95301632 bytes 2024-09-16T23:21:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 149526528 bytes 2024-09-16T23:21:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 196791296 bytes 2024-09-16T23:21:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 276317184 bytes 2024-09-16T23:21:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 284404736 bytes 2024-09-16T23:21:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 321531904 bytes 2024-09-16T23:21:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 341391360 bytes 2024-09-16T23:21:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 390975488 bytes 2024-09-16T23:21:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 396085248 bytes 2024-09-16T23:21:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 449175552 bytes 2024-09-16T23:21:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 515789824 bytes 2024-09-16T23:21:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 550299648 bytes 2024-09-16T23:21:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 579782656 bytes 2024-09-16T23:21:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 596227072 bytes 2024-09-16T23:21:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 669965312 bytes 2024-09-16T23:21:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 744248320 bytes 2024-09-16T23:21:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 768306176 bytes 2024-09-16T23:21:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 814977024 bytes 2024-09-16T23:21:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 917655552 bytes 2024-09-16T23:21:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 960065536 bytes 2024-09-16T23:21:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1037660160 bytes 2024-09-16T23:21:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1077283840 bytes 2024-09-16T23:21:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1096176640 bytes 2024-09-16T23:21:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1107720192 bytes 2024-09-16T23:21:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1108937728 bytes 2024-09-16T23:21:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1119305728 bytes 2024-09-16T23:21:30.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1189053440 bytes 2024-09-16T23:21:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1274219520 bytes 2024-09-16T23:21:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1332099072 bytes 2024-09-16T23:21:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1386676224 bytes 2024-09-16T23:21:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1430145024 bytes 2024-09-16T23:21:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1474842624 bytes 2024-09-16T23:21:31.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1530900480 bytes 2024-09-16T23:21:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1546227712 bytes 2024-09-16T23:21:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1645673472 bytes 2024-09-16T23:21:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1736756224 bytes 2024-09-16T23:21:31.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1857278976 bytes 2024-09-16T23:21:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1883146240 bytes 2024-09-16T23:21:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1924764672 bytes 2024-09-16T23:21:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1950203904 bytes 2024-09-16T23:21:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1956757504 bytes 2024-09-16T23:21:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2070328320 bytes 2024-09-16T23:21:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2113605632 bytes 2024-09-16T23:21:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2123316224 bytes 2024-09-16T23:21:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2131905536 bytes 2024-09-16T23:21:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:21:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:21:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:21:32.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_24 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:21:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:21:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-16T23:21:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:21:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-16T23:21:33.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_24': 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 139ddb7df02b 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.139ddb7df02b 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:21:33.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:32 2024 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:32 2024 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:32 2024 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 449d20a2-3295-43fc-99a8-bae9ff72cca1 2024-09-16T23:21:33.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:21:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:21:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-16T23:21:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:21:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:21:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=9 2024-09-16T23:21:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:21:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:34.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 13252608 bytes 2024-09-16T23:21:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 120911872 bytes 2024-09-16T23:21:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 139962368 bytes 2024-09-16T23:21:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 176646144 bytes 2024-09-16T23:21:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 240958464 bytes 2024-09-16T23:21:34.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 264527872 bytes 2024-09-16T23:21:34.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 361238528 bytes 2024-09-16T23:21:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 374566912 bytes 2024-09-16T23:21:35.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 420086784 bytes 2024-09-16T23:21:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 444505088 bytes 2024-09-16T23:21:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 544528384 bytes 2024-09-16T23:21:35.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 554092544 bytes 2024-09-16T23:21:35.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 596299776 bytes 2024-09-16T23:21:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 605215744 bytes 2024-09-16T23:21:35.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 613212160 bytes 2024-09-16T23:21:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 659324928 bytes 2024-09-16T23:21:36.086 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:21:36.087+0000 7f440f149640 -1 reset not still connected to 0x55f281d6b5f0 2024-09-16T23:21:36.086 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:21:36.087+0000 7f440f149640 -1 reset not still connected to 0x55f289c56680 2024-09-16T23:21:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 744262656 bytes 2024-09-16T23:21:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 774339584 bytes 2024-09-16T23:21:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 823205888 bytes 2024-09-16T23:21:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 862042112 bytes 2024-09-16T23:21:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 893337600 bytes 2024-09-16T23:21:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 925703168 bytes 2024-09-16T23:21:36.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 942894080 bytes 2024-09-16T23:21:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 981377024 bytes 2024-09-16T23:21:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 989502464 bytes 2024-09-16T23:21:37.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 995573760 bytes 2024-09-16T23:21:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1048970240 bytes 2024-09-16T23:21:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1128047616 bytes 2024-09-16T23:21:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1250895872 bytes 2024-09-16T23:21:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1331696640 bytes 2024-09-16T23:21:37.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1380389888 bytes 2024-09-16T23:21:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1415964672 bytes 2024-09-16T23:21:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1502629888 bytes 2024-09-16T23:21:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1503222784 bytes 2024-09-16T23:21:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1518147584 bytes 2024-09-16T23:21:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1592782848 bytes 2024-09-16T23:21:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1634748416 bytes 2024-09-16T23:21:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1668111360 bytes 2024-09-16T23:21:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1715309568 bytes 2024-09-16T23:21:38.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1749536768 bytes 2024-09-16T23:21:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1750153216 bytes 2024-09-16T23:21:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1775266816 bytes 2024-09-16T23:21:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1872690176 bytes 2024-09-16T23:21:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1892811776 bytes 2024-09-16T23:21:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1923200000 bytes 2024-09-16T23:21:39.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1987188736 bytes 2024-09-16T23:21:39.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2025509888 bytes 2024-09-16T23:21:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2076585984 bytes 2024-09-16T23:21:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2092542976 bytes 2024-09-16T23:21:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2094845952 bytes 2024-09-16T23:21:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2137715712 bytes 2024-09-16T23:21:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2231768064 bytes 2024-09-16T23:21:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2269963264 bytes 2024-09-16T23:21:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2305030144 bytes 2024-09-16T23:21:40.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2372512768 bytes 2024-09-16T23:21:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2390879232 bytes 2024-09-16T23:21:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2396274688 bytes 2024-09-16T23:21:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2474592256 bytes 2024-09-16T23:21:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2521710592 bytes 2024-09-16T23:21:41.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2577379328 bytes 2024-09-16T23:21:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2619883520 bytes 2024-09-16T23:21:41.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2721492992 bytes 2024-09-16T23:21:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2828465152 bytes 2024-09-16T23:21:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2829083648 bytes 2024-09-16T23:21:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2884627456 bytes 2024-09-16T23:21:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2951925760 bytes 2024-09-16T23:21:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2960715776 bytes 2024-09-16T23:21:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2993272832 bytes 2024-09-16T23:21:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3066272768 bytes 2024-09-16T23:21:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:21:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_25 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2024-09-16T23:21:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:21:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-16T23:21:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:21:43.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_25': 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13a8694f74eb 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13a8694f74eb 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:42 2024 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:42 2024 2024-09-16T23:21:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:42 2024 2024-09-16T23:21:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:21:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:21:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: aa545518-746e-469d-a87e-2348440b9692 2024-09-16T23:21:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:21:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:21:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:21:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:21:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 80549888 bytes 2024-09-16T23:21:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 105594880 bytes 2024-09-16T23:21:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 216654848 bytes 2024-09-16T23:21:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 265724928 bytes 2024-09-16T23:21:44.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 392478720 bytes 2024-09-16T23:21:44.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 495962112 bytes 2024-09-16T23:21:45.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 523956224 bytes 2024-09-16T23:21:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 592970752 bytes 2024-09-16T23:21:45.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 711311360 bytes 2024-09-16T23:21:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 742864896 bytes 2024-09-16T23:21:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 793322496 bytes 2024-09-16T23:21:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 810781696 bytes 2024-09-16T23:21:45.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 841413632 bytes 2024-09-16T23:21:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 905278464 bytes 2024-09-16T23:21:46.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 999966720 bytes 2024-09-16T23:21:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 1062733824 bytes 2024-09-16T23:21:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 1077043200 bytes 2024-09-16T23:21:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 1138309120 bytes 2024-09-16T23:21:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 1220119552 bytes 2024-09-16T23:21:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1284665344 bytes 2024-09-16T23:21:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1337338880 bytes 2024-09-16T23:21:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1401870336 bytes 2024-09-16T23:21:47.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1419304960 bytes 2024-09-16T23:21:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1499172864 bytes 2024-09-16T23:21:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1573111808 bytes 2024-09-16T23:21:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1573730304 bytes 2024-09-16T23:21:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1605053440 bytes 2024-09-16T23:21:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1660137472 bytes 2024-09-16T23:21:47.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1703677952 bytes 2024-09-16T23:21:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1738079232 bytes 2024-09-16T23:21:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1786778624 bytes 2024-09-16T23:21:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1878840320 bytes 2024-09-16T23:21:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1949713408 bytes 2024-09-16T23:21:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1984798720 bytes 2024-09-16T23:21:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 2017534976 bytes 2024-09-16T23:21:48.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 2043251712 bytes 2024-09-16T23:21:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 2080579584 bytes 2024-09-16T23:21:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 2131779584 bytes 2024-09-16T23:21:49.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 2169846784 bytes 2024-09-16T23:21:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 2195569664 bytes 2024-09-16T23:21:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 2223792128 bytes 2024-09-16T23:21:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2269507584 bytes 2024-09-16T23:21:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2282801152 bytes 2024-09-16T23:21:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2355111936 bytes 2024-09-16T23:21:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2378729472 bytes 2024-09-16T23:21:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2403244032 bytes 2024-09-16T23:21:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2443842560 bytes 2024-09-16T23:21:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2444481536 bytes 2024-09-16T23:21:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2481412096 bytes 2024-09-16T23:21:50.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2491834368 bytes 2024-09-16T23:21:50.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2546293760 bytes 2024-09-16T23:21:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2613798912 bytes 2024-09-16T23:21:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2638285824 bytes 2024-09-16T23:21:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2712970240 bytes 2024-09-16T23:21:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2727556096 bytes 2024-09-16T23:21:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2789548032 bytes 2024-09-16T23:21:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2810306560 bytes 2024-09-16T23:21:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2811862016 bytes 2024-09-16T23:21:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2874920960 bytes 2024-09-16T23:21:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2929465344 bytes 2024-09-16T23:21:51.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2986085376 bytes 2024-09-16T23:21:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 3038696448 bytes 2024-09-16T23:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_26 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-16T23:21:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:21:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:21:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:21:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-16T23:21:52.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:21:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_26': 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13b4a17ce12e 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13b4a17ce12e 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:21:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:52 2024 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:52 2024 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:52 2024 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3a054c79-09d1-4ef0-b9c5-083578f40968 2024-09-16T23:21:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:21:52.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-16T23:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:21:52.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:21:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:21:52.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:21:53.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 83772416 bytes 2024-09-16T23:21:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 117482496 bytes 2024-09-16T23:21:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 202133504 bytes 2024-09-16T23:21:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 295511040 bytes 2024-09-16T23:21:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 319639552 bytes 2024-09-16T23:21:53.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 394361856 bytes 2024-09-16T23:21:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 407493632 bytes 2024-09-16T23:21:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 451830784 bytes 2024-09-16T23:21:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 486842368 bytes 2024-09-16T23:21:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 531539968 bytes 2024-09-16T23:21:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 581498880 bytes 2024-09-16T23:21:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 632436736 bytes 2024-09-16T23:21:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 642340864 bytes 2024-09-16T23:21:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 658657280 bytes 2024-09-16T23:21:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 728360960 bytes 2024-09-16T23:21:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 773420032 bytes 2024-09-16T23:21:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 808465408 bytes 2024-09-16T23:21:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 872242176 bytes 2024-09-16T23:21:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 908755968 bytes 2024-09-16T23:21:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 927859712 bytes 2024-09-16T23:21:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 968258560 bytes 2024-09-16T23:21:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1033481216 bytes 2024-09-16T23:21:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1077364736 bytes 2024-09-16T23:21:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1126162432 bytes 2024-09-16T23:21:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1153599488 bytes 2024-09-16T23:21:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1253272576 bytes 2024-09-16T23:21:56.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1348471808 bytes 2024-09-16T23:21:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1412605952 bytes 2024-09-16T23:21:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1516094464 bytes 2024-09-16T23:21:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1516720128 bytes 2024-09-16T23:21:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1559476224 bytes 2024-09-16T23:21:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1562598400 bytes 2024-09-16T23:21:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1624083456 bytes 2024-09-16T23:21:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1713652736 bytes 2024-09-16T23:21:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1772982272 bytes 2024-09-16T23:21:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1805540352 bytes 2024-09-16T23:21:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1843404800 bytes 2024-09-16T23:21:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1902904320 bytes 2024-09-16T23:21:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1927431168 bytes 2024-09-16T23:21:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1958984704 bytes 2024-09-16T23:21:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1992978432 bytes 2024-09-16T23:21:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1997008896 bytes 2024-09-16T23:21:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2023081984 bytes 2024-09-16T23:21:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2054134784 bytes 2024-09-16T23:21:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2164399104 bytes 2024-09-16T23:21:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2220008448 bytes 2024-09-16T23:21:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2291833856 bytes 2024-09-16T23:21:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2350485504 bytes 2024-09-16T23:21:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2418359296 bytes 2024-09-16T23:21:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2478088192 bytes 2024-09-16T23:21:59.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2504289280 bytes 2024-09-16T23:21:59.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2539656192 bytes 2024-09-16T23:21:59.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2551475200 bytes 2024-09-16T23:22:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2572573696 bytes 2024-09-16T23:22:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2581575680 bytes 2024-09-16T23:22:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2595753984 bytes 2024-09-16T23:22:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2603807744 bytes 2024-09-16T23:22:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2648382464 bytes 2024-09-16T23:22:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2671931392 bytes 2024-09-16T23:22:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2722340864 bytes 2024-09-16T23:22:00.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2763797504 bytes 2024-09-16T23:22:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2812495872 bytes 2024-09-16T23:22:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2851911680 bytes 2024-09-16T23:22:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2869049344 bytes 2024-09-16T23:22:01.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2911065088 bytes 2024-09-16T23:22:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2937112576 bytes 2024-09-16T23:22:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2960482304 bytes 2024-09-16T23:22:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3016409088 bytes 2024-09-16T23:22:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3094513664 bytes 2024-09-16T23:22:01.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3162249216 bytes 2024-09-16T23:22:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3202585600 bytes 2024-09-16T23:22:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3214589952 bytes 2024-09-16T23:22:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3260351488 bytes 2024-09-16T23:22:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3272628224 bytes 2024-09-16T23:22:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3347977216 bytes 2024-09-16T23:22:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3348591616 bytes 2024-09-16T23:22:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3393649664 bytes 2024-09-16T23:22:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3415445504 bytes 2024-09-16T23:22:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3466436608 bytes 2024-09-16T23:22:03.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3513944064 bytes 2024-09-16T23:22:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3515420672 bytes 2024-09-16T23:22:03.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3543829504 bytes 2024-09-16T23:22:03.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3604222976 bytes 2024-09-16T23:22:03.536 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:22:03.538+0000 7f440f149640 -1 reset not still connected to 0x55f26b103860 2024-09-16T23:22:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3713373184 bytes 2024-09-16T23:22:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3777491968 bytes 2024-09-16T23:22:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3778087936 bytes 2024-09-16T23:22:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3799927808 bytes 2024-09-16T23:22:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3896699904 bytes 2024-09-16T23:22:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3916612608 bytes 2024-09-16T23:22:04.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9009689443802421) > weight (0.25) 2024-09-16T23:22:04.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4867641418759753) > weight (0.25) 2024-09-16T23:22:04.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7897248607618457) > weight (0.25) 2024-09-16T23:22:04.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8174012254849614) > weight (0.25) 2024-09-16T23:22:04.453 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2024-09-16T23:22:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3973957632 bytes 2024-09-16T23:22:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4002024448 bytes 2024-09-16T23:22:04.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4097434624 bytes 2024-09-16T23:22:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4165096448 bytes 2024-09-16T23:22:04.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_27 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:22:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:04.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2024-09-16T23:22:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-16T23:22:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_27': 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13c09752bc96 2024-09-16T23:22:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13c09752bc96 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:05 2024 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:05 2024 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:05 2024 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 0de0d1ea-4ea9-47eb-a8fc-cc4b4c55433f 2024-09-16T23:22:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:22:05.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:22:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 27586560 bytes 2024-09-16T23:22:06.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 77075456 bytes 2024-09-16T23:22:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 77697024 bytes 2024-09-16T23:22:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 178430976 bytes 2024-09-16T23:22:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 221837312 bytes 2024-09-16T23:22:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 280639488 bytes 2024-09-16T23:22:06.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 335328256 bytes 2024-09-16T23:22:07.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 407992320 bytes 2024-09-16T23:22:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 480976896 bytes 2024-09-16T23:22:07.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 485073920 bytes 2024-09-16T23:22:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 532972544 bytes 2024-09-16T23:22:07.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 575113216 bytes 2024-09-16T23:22:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 616821760 bytes 2024-09-16T23:22:07.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 659594240 bytes 2024-09-16T23:22:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 717395968 bytes 2024-09-16T23:22:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 770708480 bytes 2024-09-16T23:22:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 787196928 bytes 2024-09-16T23:22:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 818136064 bytes 2024-09-16T23:22:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 855575552 bytes 2024-09-16T23:22:08.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 895426560 bytes 2024-09-16T23:22:08.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 896075776 bytes 2024-09-16T23:22:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 964958208 bytes 2024-09-16T23:22:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 986522624 bytes 2024-09-16T23:22:09.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1034599424 bytes 2024-09-16T23:22:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1081963520 bytes 2024-09-16T23:22:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1126506496 bytes 2024-09-16T23:22:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1150705664 bytes 2024-09-16T23:22:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1171265536 bytes 2024-09-16T23:22:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1196256256 bytes 2024-09-16T23:22:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1230587904 bytes 2024-09-16T23:22:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1254861824 bytes 2024-09-16T23:22:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1280559104 bytes 2024-09-16T23:22:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1302673408 bytes 2024-09-16T23:22:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1398504448 bytes 2024-09-16T23:22:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1418075136 bytes 2024-09-16T23:22:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1462604800 bytes 2024-09-16T23:22:10.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1505120256 bytes 2024-09-16T23:22:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1654252544 bytes 2024-09-16T23:22:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1681201152 bytes 2024-09-16T23:22:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1701839872 bytes 2024-09-16T23:22:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1730719744 bytes 2024-09-16T23:22:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1789297664 bytes 2024-09-16T23:22:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1801908224 bytes 2024-09-16T23:22:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1858026496 bytes 2024-09-16T23:22:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1858656256 bytes 2024-09-16T23:22:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1899569152 bytes 2024-09-16T23:22:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1974328320 bytes 2024-09-16T23:22:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2066649088 bytes 2024-09-16T23:22:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2067273728 bytes 2024-09-16T23:22:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2092981248 bytes 2024-09-16T23:22:12.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2172232704 bytes 2024-09-16T23:22:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2280909824 bytes 2024-09-16T23:22:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2289645568 bytes 2024-09-16T23:22:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2340602880 bytes 2024-09-16T23:22:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2434540544 bytes 2024-09-16T23:22:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2435151872 bytes 2024-09-16T23:22:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2534183936 bytes 2024-09-16T23:22:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2571910144 bytes 2024-09-16T23:22:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2607005696 bytes 2024-09-16T23:22:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2624890880 bytes 2024-09-16T23:22:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2682555392 bytes 2024-09-16T23:22:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2723186688 bytes 2024-09-16T23:22:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2742344704 bytes 2024-09-16T23:22:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_28 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:22:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-16T23:22:14.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-16T23:22:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_28': 2024-09-16T23:22:15.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13ccaeac3aaf 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13ccaeac3aaf 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:14 2024 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:14 2024 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:14 2024 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 77ea12b7-17e8-4b4a-ad76-3f55bd6a6279 2024-09-16T23:22:15.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:22:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=11 2024-09-16T23:22:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 32540672 bytes 2024-09-16T23:22:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 108622848 bytes 2024-09-16T23:22:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 117868544 bytes 2024-09-16T23:22:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 205829120 bytes 2024-09-16T23:22:15.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 211405824 bytes 2024-09-16T23:22:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 234257408 bytes 2024-09-16T23:22:15.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 275991552 bytes 2024-09-16T23:22:16.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 338600960 bytes 2024-09-16T23:22:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 339218432 bytes 2024-09-16T23:22:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 437614592 bytes 2024-09-16T23:22:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 494822400 bytes 2024-09-16T23:22:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 567286784 bytes 2024-09-16T23:22:16.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 606244864 bytes 2024-09-16T23:22:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 694422528 bytes 2024-09-16T23:22:17.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 701429760 bytes 2024-09-16T23:22:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 726625280 bytes 2024-09-16T23:22:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 727235584 bytes 2024-09-16T23:22:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 757980160 bytes 2024-09-16T23:22:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 768408576 bytes 2024-09-16T23:22:17.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 817640448 bytes 2024-09-16T23:22:17.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 869096448 bytes 2024-09-16T23:22:17.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 939464704 bytes 2024-09-16T23:22:18.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 979365888 bytes 2024-09-16T23:22:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1002993664 bytes 2024-09-16T23:22:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1014020096 bytes 2024-09-16T23:22:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1072295936 bytes 2024-09-16T23:22:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1103100928 bytes 2024-09-16T23:22:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1199264768 bytes 2024-09-16T23:22:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1281183744 bytes 2024-09-16T23:22:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1296122880 bytes 2024-09-16T23:22:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1304919040 bytes 2024-09-16T23:22:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1374514176 bytes 2024-09-16T23:22:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1401743360 bytes 2024-09-16T23:22:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1427806208 bytes 2024-09-16T23:22:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1472957440 bytes 2024-09-16T23:22:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1535076352 bytes 2024-09-16T23:22:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1629359104 bytes 2024-09-16T23:22:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1652170752 bytes 2024-09-16T23:22:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1671706624 bytes 2024-09-16T23:22:20.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1713060864 bytes 2024-09-16T23:22:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1741227008 bytes 2024-09-16T23:22:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1784844288 bytes 2024-09-16T23:22:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1829253120 bytes 2024-09-16T23:22:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1897250816 bytes 2024-09-16T23:22:20.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1989220352 bytes 2024-09-16T23:22:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2014781440 bytes 2024-09-16T23:22:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2062751744 bytes 2024-09-16T23:22:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2143115264 bytes 2024-09-16T23:22:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2189283328 bytes 2024-09-16T23:22:21.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2245487616 bytes 2024-09-16T23:22:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2247510016 bytes 2024-09-16T23:22:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2287369216 bytes 2024-09-16T23:22:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2323563520 bytes 2024-09-16T23:22:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2405719040 bytes 2024-09-16T23:22:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2438124544 bytes 2024-09-16T23:22:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2451298304 bytes 2024-09-16T23:22:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2517707776 bytes 2024-09-16T23:22:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2538551296 bytes 2024-09-16T23:22:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2578996224 bytes 2024-09-16T23:22:22.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2626420736 bytes 2024-09-16T23:22:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2671534080 bytes 2024-09-16T23:22:23.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2737007616 bytes 2024-09-16T23:22:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2802932736 bytes 2024-09-16T23:22:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2849225728 bytes 2024-09-16T23:22:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2898221056 bytes 2024-09-16T23:22:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2933696512 bytes 2024-09-16T23:22:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2953648128 bytes 2024-09-16T23:22:23.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3022420992 bytes 2024-09-16T23:22:24.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3081931776 bytes 2024-09-16T23:22:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3110330368 bytes 2024-09-16T23:22:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3134551040 bytes 2024-09-16T23:22:24.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3145328640 bytes 2024-09-16T23:22:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3238843392 bytes 2024-09-16T23:22:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3284222976 bytes 2024-09-16T23:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3324807168 bytes 2024-09-16T23:22:24.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3332242432 bytes 2024-09-16T23:22:24.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3370552320 bytes 2024-09-16T23:22:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3371173888 bytes 2024-09-16T23:22:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3396667392 bytes 2024-09-16T23:22:25.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3420838912 bytes 2024-09-16T23:22:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3424351232 bytes 2024-09-16T23:22:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3448572928 bytes 2024-09-16T23:22:25.495 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:22:25.497+0000 7f440f149640 -1 reset not still connected to 0x55f2674c48f0 2024-09-16T23:22:25.495 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:22:25.497+0000 7f440f149640 -1 reset not still connected to 0x55f2800b6c30 2024-09-16T23:22:25.495 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:22:25.498+0000 7f440f149640 -1 reset not still connected to 0x55f2800b7e10 2024-09-16T23:22:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3477921792 bytes 2024-09-16T23:22:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3522229248 bytes 2024-09-16T23:22:25.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3545604096 bytes 2024-09-16T23:22:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3568121856 bytes 2024-09-16T23:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:26.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_29 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:26.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2024-09-16T23:22:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-16T23:22:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_29': 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13d89e98d5c 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13d89e98d5c 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:26 2024 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:26 2024 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:26 2024 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: d119fabc-3461-4f14-ad5b-937b0502300c 2024-09-16T23:22:27.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-16T23:22:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:22:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:22:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 70508544 bytes 2024-09-16T23:22:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 130814976 bytes 2024-09-16T23:22:27.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 200927232 bytes 2024-09-16T23:22:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 228626432 bytes 2024-09-16T23:22:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 249123840 bytes 2024-09-16T23:22:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 258984960 bytes 2024-09-16T23:22:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 308667392 bytes 2024-09-16T23:22:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 319109120 bytes 2024-09-16T23:22:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 430745600 bytes 2024-09-16T23:22:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 494748672 bytes 2024-09-16T23:22:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 615384064 bytes 2024-09-16T23:22:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 658049024 bytes 2024-09-16T23:22:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 674956288 bytes 2024-09-16T23:22:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 730041344 bytes 2024-09-16T23:22:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 745379840 bytes 2024-09-16T23:22:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 746866688 bytes 2024-09-16T23:22:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 793923584 bytes 2024-09-16T23:22:29.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 817024000 bytes 2024-09-16T23:22:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 882433024 bytes 2024-09-16T23:22:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 918330368 bytes 2024-09-16T23:22:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 944436224 bytes 2024-09-16T23:22:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1006206976 bytes 2024-09-16T23:22:29.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1040009216 bytes 2024-09-16T23:22:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1119210496 bytes 2024-09-16T23:22:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1152377856 bytes 2024-09-16T23:22:30.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1186582528 bytes 2024-09-16T23:22:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1209795584 bytes 2024-09-16T23:22:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1280422912 bytes 2024-09-16T23:22:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1315185664 bytes 2024-09-16T23:22:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1340560384 bytes 2024-09-16T23:22:30.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1383828480 bytes 2024-09-16T23:22:31.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1416116224 bytes 2024-09-16T23:22:31.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1460000768 bytes 2024-09-16T23:22:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1500740608 bytes 2024-09-16T23:22:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1534619648 bytes 2024-09-16T23:22:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1535230976 bytes 2024-09-16T23:22:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1565262848 bytes 2024-09-16T23:22:31.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1567491072 bytes 2024-09-16T23:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1602453504 bytes 2024-09-16T23:22:31.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1630892032 bytes 2024-09-16T23:22:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1631496192 bytes 2024-09-16T23:22:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1650953216 bytes 2024-09-16T23:22:32.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1682539520 bytes 2024-09-16T23:22:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1715299328 bytes 2024-09-16T23:22:32.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1735677952 bytes 2024-09-16T23:22:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1789708288 bytes 2024-09-16T23:22:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1806588928 bytes 2024-09-16T23:22:32.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1873800192 bytes 2024-09-16T23:22:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1902532608 bytes 2024-09-16T23:22:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1974719488 bytes 2024-09-16T23:22:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2040765440 bytes 2024-09-16T23:22:33.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2084250624 bytes 2024-09-16T23:22:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2125180928 bytes 2024-09-16T23:22:33.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2163804160 bytes 2024-09-16T23:22:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2222338048 bytes 2024-09-16T23:22:33.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2358333440 bytes 2024-09-16T23:22:34.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2368067584 bytes 2024-09-16T23:22:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_30 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:34.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2024-09-16T23:22:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:34.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:34.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-16T23:22:34.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-16T23:22:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_30': 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13e49e370cdb 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13e49e370cdb 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:34 2024 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:34 2024 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:34 2024 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 9a9a2e05-0891-4a2c-bb16-07ba21d8071f 2024-09-16T23:22:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-16T23:22:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:22:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:22:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 18076672 bytes 2024-09-16T23:22:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 51443712 bytes 2024-09-16T23:22:35.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 127011840 bytes 2024-09-16T23:22:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 135316480 bytes 2024-09-16T23:22:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 157426688 bytes 2024-09-16T23:22:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 167025664 bytes 2024-09-16T23:22:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 197533696 bytes 2024-09-16T23:22:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 247848960 bytes 2024-09-16T23:22:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 275972096 bytes 2024-09-16T23:22:36.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 334684160 bytes 2024-09-16T23:22:36.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 401318912 bytes 2024-09-16T23:22:36.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 439207936 bytes 2024-09-16T23:22:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 482423808 bytes 2024-09-16T23:22:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 530139136 bytes 2024-09-16T23:22:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 576905216 bytes 2024-09-16T23:22:36.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 602243072 bytes 2024-09-16T23:22:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 614663168 bytes 2024-09-16T23:22:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 685560832 bytes 2024-09-16T23:22:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 686149632 bytes 2024-09-16T23:22:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 686782464 bytes 2024-09-16T23:22:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 690969600 bytes 2024-09-16T23:22:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 694902784 bytes 2024-09-16T23:22:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 787119104 bytes 2024-09-16T23:22:37.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 907831296 bytes 2024-09-16T23:22:37.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 947766272 bytes 2024-09-16T23:22:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 971283456 bytes 2024-09-16T23:22:38.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1041506304 bytes 2024-09-16T23:22:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1092708352 bytes 2024-09-16T23:22:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1099815936 bytes 2024-09-16T23:22:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1166755840 bytes 2024-09-16T23:22:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1205426176 bytes 2024-09-16T23:22:38.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1240848384 bytes 2024-09-16T23:22:38.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1285311488 bytes 2024-09-16T23:22:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1359498240 bytes 2024-09-16T23:22:39.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1376212992 bytes 2024-09-16T23:22:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1401928704 bytes 2024-09-16T23:22:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1444791296 bytes 2024-09-16T23:22:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1524046848 bytes 2024-09-16T23:22:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1595433984 bytes 2024-09-16T23:22:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1613143040 bytes 2024-09-16T23:22:39.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1676091392 bytes 2024-09-16T23:22:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1683838976 bytes 2024-09-16T23:22:40.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1715453952 bytes 2024-09-16T23:22:40.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1749017600 bytes 2024-09-16T23:22:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1750080512 bytes 2024-09-16T23:22:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1818912768 bytes 2024-09-16T23:22:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1889604608 bytes 2024-09-16T23:22:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1962877952 bytes 2024-09-16T23:22:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1977863168 bytes 2024-09-16T23:22:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2034289664 bytes 2024-09-16T23:22:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2048432128 bytes 2024-09-16T23:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2117350400 bytes 2024-09-16T23:22:41.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2154025984 bytes 2024-09-16T23:22:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2165362688 bytes 2024-09-16T23:22:41.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2202149888 bytes 2024-09-16T23:22:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2226536448 bytes 2024-09-16T23:22:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2310799360 bytes 2024-09-16T23:22:41.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2335644672 bytes 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:42.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_31 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-16T23:22:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2024-09-16T23:22:42.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-16T23:22:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_31': 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13f0625efccd 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13f0625efccd 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:42 2024 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:42 2024 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:42 2024 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c6104db7-e7bf-4eea-a889-773f358930c7 2024-09-16T23:22:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:22:43.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:22:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:43.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 53184512 bytes 2024-09-16T23:22:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 107377664 bytes 2024-09-16T23:22:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 119894016 bytes 2024-09-16T23:22:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 181303296 bytes 2024-09-16T23:22:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 216538112 bytes 2024-09-16T23:22:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 272694272 bytes 2024-09-16T23:22:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 291520512 bytes 2024-09-16T23:22:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 325566464 bytes 2024-09-16T23:22:44.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 365224960 bytes 2024-09-16T23:22:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 424760320 bytes 2024-09-16T23:22:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 448846848 bytes 2024-09-16T23:22:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 464957440 bytes 2024-09-16T23:22:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 588001280 bytes 2024-09-16T23:22:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 612560896 bytes 2024-09-16T23:22:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 679618560 bytes 2024-09-16T23:22:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 709474304 bytes 2024-09-16T23:22:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 735523840 bytes 2024-09-16T23:22:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 784726016 bytes 2024-09-16T23:22:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 815781888 bytes 2024-09-16T23:22:45.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 834848768 bytes 2024-09-16T23:22:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 927688704 bytes 2024-09-16T23:22:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 953681920 bytes 2024-09-16T23:22:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1013467136 bytes 2024-09-16T23:22:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1092298752 bytes 2024-09-16T23:22:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1126075392 bytes 2024-09-16T23:22:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1148451840 bytes 2024-09-16T23:22:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1186165760 bytes 2024-09-16T23:22:46.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1255292928 bytes 2024-09-16T23:22:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1291074560 bytes 2024-09-16T23:22:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1353789440 bytes 2024-09-16T23:22:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1393757184 bytes 2024-09-16T23:22:47.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1444052992 bytes 2024-09-16T23:22:47.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1484043264 bytes 2024-09-16T23:22:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1514105856 bytes 2024-09-16T23:22:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1564271616 bytes 2024-09-16T23:22:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1658064896 bytes 2024-09-16T23:22:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1720017920 bytes 2024-09-16T23:22:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1753441280 bytes 2024-09-16T23:22:48.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1817406464 bytes 2024-09-16T23:22:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1897099264 bytes 2024-09-16T23:22:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1954849792 bytes 2024-09-16T23:22:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1978935296 bytes 2024-09-16T23:22:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2049147904 bytes 2024-09-16T23:22:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2091711488 bytes 2024-09-16T23:22:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2136398848 bytes 2024-09-16T23:22:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2159533056 bytes 2024-09-16T23:22:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2239273984 bytes 2024-09-16T23:22:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2317370368 bytes 2024-09-16T23:22:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2368569344 bytes 2024-09-16T23:22:49.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8822142327617084) > weight (0.25) 2024-09-16T23:22:49.365 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.32863333479113177) > weight (0.25) 2024-09-16T23:22:49.365 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T23:22:49.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T23:22:49.366 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:22:49.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:22:49.368+0000 7fc5408d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42723) UID: 1000 2024-09-16T23:22:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2424733696 bytes 2024-09-16T23:22:49.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2491436032 bytes 2024-09-16T23:22:49.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2496775168 bytes 2024-09-16T23:22:49.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2498516992 bytes 2024-09-16T23:22:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2538601472 bytes 2024-09-16T23:22:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:22:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:22:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:22:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_32 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-16T23:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:22:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:50.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:22:50.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-16T23:22:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:22:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-16T23:22:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_32': 2024-09-16T23:22:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:22:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:22:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:22:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 13fc3becb273 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.13fc3becb273 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:50 2024 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:50 2024 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:50 2024 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 11ebbc36-f469-465c-b15a-fe40b29147fb 2024-09-16T23:22:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:22:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:22:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=9 2024-09-16T23:22:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:22:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:22:51.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 31863808 bytes 2024-09-16T23:22:51.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 60713984 bytes 2024-09-16T23:22:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 121657344 bytes 2024-09-16T23:22:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 163720192 bytes 2024-09-16T23:22:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 184285184 bytes 2024-09-16T23:22:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 208638976 bytes 2024-09-16T23:22:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 281975808 bytes 2024-09-16T23:22:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 298841088 bytes 2024-09-16T23:22:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 350183424 bytes 2024-09-16T23:22:52.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 365217792 bytes 2024-09-16T23:22:52.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 413639680 bytes 2024-09-16T23:22:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 427390976 bytes 2024-09-16T23:22:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 460129280 bytes 2024-09-16T23:22:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 512157696 bytes 2024-09-16T23:22:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 550242304 bytes 2024-09-16T23:22:52.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 589003776 bytes 2024-09-16T23:22:53.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 612628480 bytes 2024-09-16T23:22:53.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 641425408 bytes 2024-09-16T23:22:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 752587776 bytes 2024-09-16T23:22:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 827426816 bytes 2024-09-16T23:22:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 895410176 bytes 2024-09-16T23:22:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 970373120 bytes 2024-09-16T23:22:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 995919872 bytes 2024-09-16T23:22:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1013454848 bytes 2024-09-16T23:22:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1090698240 bytes 2024-09-16T23:22:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1147555840 bytes 2024-09-16T23:22:54.367 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T23:22:54.367 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:22:54.368 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T23:22:54.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:22:54.368 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:22:54.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:22:54.374 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-16T23:22:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1196858368 bytes 2024-09-16T23:22:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1284832256 bytes 2024-09-16T23:22:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1342502912 bytes 2024-09-16T23:22:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1359730688 bytes 2024-09-16T23:22:55.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1438403584 bytes 2024-09-16T23:22:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1491041280 bytes 2024-09-16T23:22:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1534474240 bytes 2024-09-16T23:22:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1619476480 bytes 2024-09-16T23:22:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1637331968 bytes 2024-09-16T23:22:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1702317056 bytes 2024-09-16T23:22:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1736233984 bytes 2024-09-16T23:22:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1751915520 bytes 2024-09-16T23:22:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1787845632 bytes 2024-09-16T23:22:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1811239936 bytes 2024-09-16T23:22:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1874521088 bytes 2024-09-16T23:22:56.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1894468608 bytes 2024-09-16T23:22:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1910047744 bytes 2024-09-16T23:22:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1975106560 bytes 2024-09-16T23:22:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2046912512 bytes 2024-09-16T23:22:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2075837440 bytes 2024-09-16T23:22:56.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2141720576 bytes 2024-09-16T23:22:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2170847232 bytes 2024-09-16T23:22:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2192096256 bytes 2024-09-16T23:22:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2255708160 bytes 2024-09-16T23:22:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2272262144 bytes 2024-09-16T23:22:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2346619904 bytes 2024-09-16T23:22:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2389554176 bytes 2024-09-16T23:22:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2409799680 bytes 2024-09-16T23:22:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2438039552 bytes 2024-09-16T23:22:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2482525184 bytes 2024-09-16T23:22:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2486758400 bytes 2024-09-16T23:22:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2487374848 bytes 2024-09-16T23:22:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2502525952 bytes 2024-09-16T23:22:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2578486272 bytes 2024-09-16T23:22:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2589480960 bytes 2024-09-16T23:22:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2604390400 bytes 2024-09-16T23:22:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2656109568 bytes 2024-09-16T23:22:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2716160000 bytes 2024-09-16T23:22:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2762205184 bytes 2024-09-16T23:22:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2842076160 bytes 2024-09-16T23:22:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2879381504 bytes 2024-09-16T23:22:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2913705984 bytes 2024-09-16T23:22:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2923245568 bytes 2024-09-16T23:22:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2943048704 bytes 2024-09-16T23:22:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3001015296 bytes 2024-09-16T23:22:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3047468032 bytes 2024-09-16T23:23:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3109245952 bytes 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_33 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2024-09-16T23:23:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:00.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-16T23:23:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:23:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-16T23:23:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:01.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-16T23:23:01.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_33': 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14101a9ac8cb 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14101a9ac8cb 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:00 2024 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:00 2024 2024-09-16T23:23:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:00 2024 2024-09-16T23:23:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: e491a9c0-f499-472f-b800-fabf601f9b18 2024-09-16T23:23:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-16T23:23:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:23:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:23:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 49216512 bytes 2024-09-16T23:23:01.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 95472640 bytes 2024-09-16T23:23:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 199830528 bytes 2024-09-16T23:23:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 262943744 bytes 2024-09-16T23:23:01.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 331834368 bytes 2024-09-16T23:23:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 340402176 bytes 2024-09-16T23:23:02.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 365164544 bytes 2024-09-16T23:23:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 438718464 bytes 2024-09-16T23:23:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 458375168 bytes 2024-09-16T23:23:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 516943872 bytes 2024-09-16T23:23:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 591465472 bytes 2024-09-16T23:23:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 615053312 bytes 2024-09-16T23:23:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 678495232 bytes 2024-09-16T23:23:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 722553856 bytes 2024-09-16T23:23:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 736538624 bytes 2024-09-16T23:23:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 750765056 bytes 2024-09-16T23:23:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 751370240 bytes 2024-09-16T23:23:03.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 759056384 bytes 2024-09-16T23:23:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 802334720 bytes 2024-09-16T23:23:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 820875264 bytes 2024-09-16T23:23:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 872124416 bytes 2024-09-16T23:23:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 876847104 bytes 2024-09-16T23:23:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 927785984 bytes 2024-09-16T23:23:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 975449088 bytes 2024-09-16T23:23:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1079634944 bytes 2024-09-16T23:23:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1126219776 bytes 2024-09-16T23:23:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1172375552 bytes 2024-09-16T23:23:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1206553600 bytes 2024-09-16T23:23:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1207161856 bytes 2024-09-16T23:23:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1210249216 bytes 2024-09-16T23:23:04.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1221548032 bytes 2024-09-16T23:23:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1252877312 bytes 2024-09-16T23:23:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1271877632 bytes 2024-09-16T23:23:05.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1325950976 bytes 2024-09-16T23:23:05.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1411523584 bytes 2024-09-16T23:23:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1438620672 bytes 2024-09-16T23:23:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1510473728 bytes 2024-09-16T23:23:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1575628800 bytes 2024-09-16T23:23:05.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1612607488 bytes 2024-09-16T23:23:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1644966912 bytes 2024-09-16T23:23:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1679735808 bytes 2024-09-16T23:23:06.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1696838656 bytes 2024-09-16T23:23:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1735483392 bytes 2024-09-16T23:23:06.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1753367552 bytes 2024-09-16T23:23:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1839294464 bytes 2024-09-16T23:23:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1941484544 bytes 2024-09-16T23:23:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1960530944 bytes 2024-09-16T23:23:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1970822144 bytes 2024-09-16T23:23:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1984532480 bytes 2024-09-16T23:23:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2012360704 bytes 2024-09-16T23:23:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2074030080 bytes 2024-09-16T23:23:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2121422848 bytes 2024-09-16T23:23:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2139507712 bytes 2024-09-16T23:23:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2204020736 bytes 2024-09-16T23:23:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2248720384 bytes 2024-09-16T23:23:07.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2281094144 bytes 2024-09-16T23:23:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2300290048 bytes 2024-09-16T23:23:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2353982464 bytes 2024-09-16T23:23:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2389227520 bytes 2024-09-16T23:23:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2442342400 bytes 2024-09-16T23:23:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2535226368 bytes 2024-09-16T23:23:08.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2556130304 bytes 2024-09-16T23:23:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2604203008 bytes 2024-09-16T23:23:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2634021888 bytes 2024-09-16T23:23:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2643237888 bytes 2024-09-16T23:23:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2661685248 bytes 2024-09-16T23:23:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_34 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-16T23:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:09.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:09.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2024-09-16T23:23:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-16T23:23:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_34': 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 141ced773f49 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.141ced773f49 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:10.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:09 2024 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:09 2024 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:09 2024 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2dc3b71d-4f9f-4123-bb64-1b41606c25b5 2024-09-16T23:23:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:23:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=5 2024-09-16T23:23:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 46766080 bytes 2024-09-16T23:23:10.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 99309568 bytes 2024-09-16T23:23:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 147787776 bytes 2024-09-16T23:23:10.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 190359552 bytes 2024-09-16T23:23:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 309361664 bytes 2024-09-16T23:23:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 329086976 bytes 2024-09-16T23:23:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 369912832 bytes 2024-09-16T23:23:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 458378240 bytes 2024-09-16T23:23:11.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 482710528 bytes 2024-09-16T23:23:11.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 542200832 bytes 2024-09-16T23:23:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 570647552 bytes 2024-09-16T23:23:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 619146240 bytes 2024-09-16T23:23:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 704382976 bytes 2024-09-16T23:23:11.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 729166848 bytes 2024-09-16T23:23:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 801297408 bytes 2024-09-16T23:23:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 858082304 bytes 2024-09-16T23:23:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 902816768 bytes 2024-09-16T23:23:12.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 918074368 bytes 2024-09-16T23:23:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 945517568 bytes 2024-09-16T23:23:12.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 979184640 bytes 2024-09-16T23:23:12.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1023330304 bytes 2024-09-16T23:23:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1116741632 bytes 2024-09-16T23:23:13.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1174413312 bytes 2024-09-16T23:23:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1249890304 bytes 2024-09-16T23:23:13.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1299485696 bytes 2024-09-16T23:23:13.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1399733248 bytes 2024-09-16T23:23:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1434608640 bytes 2024-09-16T23:23:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1480399872 bytes 2024-09-16T23:23:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1508343808 bytes 2024-09-16T23:23:14.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1565218816 bytes 2024-09-16T23:23:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1616256000 bytes 2024-09-16T23:23:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1645608960 bytes 2024-09-16T23:23:14.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1778718720 bytes 2024-09-16T23:23:14.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1800502272 bytes 2024-09-16T23:23:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1851720704 bytes 2024-09-16T23:23:14.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1891445760 bytes 2024-09-16T23:23:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1896198144 bytes 2024-09-16T23:23:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1944755200 bytes 2024-09-16T23:23:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_35 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2024-09-16T23:23:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:15.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-16T23:23:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-16T23:23:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_35': 2024-09-16T23:23:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:16.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14283138d635 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14283138d635 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:15 2024 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:15 2024 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:15 2024 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: a2742cc4-6203-4618-aa8e-430f3fe6d0e9 2024-09-16T23:23:16.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:23:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:23:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 37425152 bytes 2024-09-16T23:23:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 91749376 bytes 2024-09-16T23:23:16.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 117189632 bytes 2024-09-16T23:23:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 146032640 bytes 2024-09-16T23:23:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 176461824 bytes 2024-09-16T23:23:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 209682432 bytes 2024-09-16T23:23:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 231483392 bytes 2024-09-16T23:23:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 262488064 bytes 2024-09-16T23:23:17.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 291325952 bytes 2024-09-16T23:23:17.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 338468864 bytes 2024-09-16T23:23:17.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 340441088 bytes 2024-09-16T23:23:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 364537856 bytes 2024-09-16T23:23:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 431706112 bytes 2024-09-16T23:23:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 476968960 bytes 2024-09-16T23:23:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 477587456 bytes 2024-09-16T23:23:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 511906816 bytes 2024-09-16T23:23:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 537047040 bytes 2024-09-16T23:23:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 661819392 bytes 2024-09-16T23:23:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 702334976 bytes 2024-09-16T23:23:18.562 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 848708608 bytes 2024-09-16T23:23:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 866638848 bytes 2024-09-16T23:23:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 902940672 bytes 2024-09-16T23:23:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 904685568 bytes 2024-09-16T23:23:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 930163712 bytes 2024-09-16T23:23:19.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 994003968 bytes 2024-09-16T23:23:19.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1046074368 bytes 2024-09-16T23:23:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1119838208 bytes 2024-09-16T23:23:19.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1185666048 bytes 2024-09-16T23:23:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1288880128 bytes 2024-09-16T23:23:19.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1289499648 bytes 2024-09-16T23:23:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1360399360 bytes 2024-09-16T23:23:20.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1415017472 bytes 2024-09-16T23:23:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1453873152 bytes 2024-09-16T23:23:20.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1483268096 bytes 2024-09-16T23:23:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1537351680 bytes 2024-09-16T23:23:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1549469696 bytes 2024-09-16T23:23:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1605628928 bytes 2024-09-16T23:23:20.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1711821824 bytes 2024-09-16T23:23:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1822620672 bytes 2024-09-16T23:23:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1870390272 bytes 2024-09-16T23:23:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1891200000 bytes 2024-09-16T23:23:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1940985856 bytes 2024-09-16T23:23:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1976287232 bytes 2024-09-16T23:23:21.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2025680896 bytes 2024-09-16T23:23:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2032025600 bytes 2024-09-16T23:23:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2056978432 bytes 2024-09-16T23:23:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2168098816 bytes 2024-09-16T23:23:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2191417344 bytes 2024-09-16T23:23:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2251990016 bytes 2024-09-16T23:23:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2311852032 bytes 2024-09-16T23:23:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2344907776 bytes 2024-09-16T23:23:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2349515776 bytes 2024-09-16T23:23:22.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2350146560 bytes 2024-09-16T23:23:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2381983744 bytes 2024-09-16T23:23:22.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2384596992 bytes 2024-09-16T23:23:23.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2409650176 bytes 2024-09-16T23:23:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2457362432 bytes 2024-09-16T23:23:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2466545664 bytes 2024-09-16T23:23:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2492980224 bytes 2024-09-16T23:23:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2591290368 bytes 2024-09-16T23:23:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2626509824 bytes 2024-09-16T23:23:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2707040256 bytes 2024-09-16T23:23:23.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2827761664 bytes 2024-09-16T23:23:24.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2868619264 bytes 2024-09-16T23:23:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2934763520 bytes 2024-09-16T23:23:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2987656192 bytes 2024-09-16T23:23:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3011737600 bytes 2024-09-16T23:23:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3045953536 bytes 2024-09-16T23:23:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3064608768 bytes 2024-09-16T23:23:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3109129216 bytes 2024-09-16T23:23:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3125055488 bytes 2024-09-16T23:23:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3126884352 bytes 2024-09-16T23:23:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3195908096 bytes 2024-09-16T23:23:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3281385472 bytes 2024-09-16T23:23:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3372692480 bytes 2024-09-16T23:23:25.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3460586496 bytes 2024-09-16T23:23:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3554546688 bytes 2024-09-16T23:23:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3604326400 bytes 2024-09-16T23:23:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3633644544 bytes 2024-09-16T23:23:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_36 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2024-09-16T23:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-16T23:23:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:23:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-16T23:23:27.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:27.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-16T23:23:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_36': 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14347023485c 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14347023485c 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:26 2024 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:26 2024 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:26 2024 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 03dea910-3ab0-46f2-a90e-2ddda005fc50 2024-09-16T23:23:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:23:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:23:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:27.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:27.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 52624384 bytes 2024-09-16T23:23:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 107022336 bytes 2024-09-16T23:23:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 164481024 bytes 2024-09-16T23:23:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 232929280 bytes 2024-09-16T23:23:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 312949760 bytes 2024-09-16T23:23:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 387892224 bytes 2024-09-16T23:23:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 394009600 bytes 2024-09-16T23:23:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 417981440 bytes 2024-09-16T23:23:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 439749632 bytes 2024-09-16T23:23:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 440372224 bytes 2024-09-16T23:23:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 465695744 bytes 2024-09-16T23:23:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 497165312 bytes 2024-09-16T23:23:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 554159104 bytes 2024-09-16T23:23:29.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 616225792 bytes 2024-09-16T23:23:29.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 627075072 bytes 2024-09-16T23:23:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 669643776 bytes 2024-09-16T23:23:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 713155584 bytes 2024-09-16T23:23:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 758242304 bytes 2024-09-16T23:23:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 823875584 bytes 2024-09-16T23:23:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 873156608 bytes 2024-09-16T23:23:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 959876096 bytes 2024-09-16T23:23:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1052805120 bytes 2024-09-16T23:23:30.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1106359296 bytes 2024-09-16T23:23:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1186041856 bytes 2024-09-16T23:23:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1268767744 bytes 2024-09-16T23:23:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1362209792 bytes 2024-09-16T23:23:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1419195392 bytes 2024-09-16T23:23:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1434353664 bytes 2024-09-16T23:23:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1478057984 bytes 2024-09-16T23:23:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1478677504 bytes 2024-09-16T23:23:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1578441728 bytes 2024-09-16T23:23:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1657784320 bytes 2024-09-16T23:23:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1712403456 bytes 2024-09-16T23:23:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1765918720 bytes 2024-09-16T23:23:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1767810048 bytes 2024-09-16T23:23:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1839984640 bytes 2024-09-16T23:23:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1864830976 bytes 2024-09-16T23:23:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1881001984 bytes 2024-09-16T23:23:32.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1952765952 bytes 2024-09-16T23:23:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 2024791040 bytes 2024-09-16T23:23:32.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 2034203648 bytes 2024-09-16T23:23:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2065486848 bytes 2024-09-16T23:23:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2089176064 bytes 2024-09-16T23:23:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2136086528 bytes 2024-09-16T23:23:33.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2211942400 bytes 2024-09-16T23:23:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2246858752 bytes 2024-09-16T23:23:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2319201280 bytes 2024-09-16T23:23:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2343694336 bytes 2024-09-16T23:23:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2374504448 bytes 2024-09-16T23:23:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2419451904 bytes 2024-09-16T23:23:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2451209216 bytes 2024-09-16T23:23:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2498587648 bytes 2024-09-16T23:23:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2567416832 bytes 2024-09-16T23:23:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2568055808 bytes 2024-09-16T23:23:34.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2617323520 bytes 2024-09-16T23:23:34.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2740578304 bytes 2024-09-16T23:23:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2790422528 bytes 2024-09-16T23:23:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2832245760 bytes 2024-09-16T23:23:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2872472576 bytes 2024-09-16T23:23:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2890135552 bytes 2024-09-16T23:23:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2934836224 bytes 2024-09-16T23:23:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2973162496 bytes 2024-09-16T23:23:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2978467840 bytes 2024-09-16T23:23:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3067063296 bytes 2024-09-16T23:23:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3114601472 bytes 2024-09-16T23:23:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3133068288 bytes 2024-09-16T23:23:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3218000896 bytes 2024-09-16T23:23:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3306420224 bytes 2024-09-16T23:23:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3374047232 bytes 2024-09-16T23:23:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3411504128 bytes 2024-09-16T23:23:36.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3449767936 bytes 2024-09-16T23:23:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3456743424 bytes 2024-09-16T23:23:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3481784320 bytes 2024-09-16T23:23:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3505296384 bytes 2024-09-16T23:23:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3558340608 bytes 2024-09-16T23:23:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3630186496 bytes 2024-09-16T23:23:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3662149632 bytes 2024-09-16T23:23:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3700412416 bytes 2024-09-16T23:23:37.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3777220608 bytes 2024-09-16T23:23:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3851403264 bytes 2024-09-16T23:23:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3875990528 bytes 2024-09-16T23:23:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3920820224 bytes 2024-09-16T23:23:37.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3925544960 bytes 2024-09-16T23:23:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3976826880 bytes 2024-09-16T23:23:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 4018134016 bytes 2024-09-16T23:23:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 4049431552 bytes 2024-09-16T23:23:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 4069432320 bytes 2024-09-16T23:23:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 4097046528 bytes 2024-09-16T23:23:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 4186446848 bytes 2024-09-16T23:23:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4279183360 bytes 2024-09-16T23:23:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4288447488 bytes 2024-09-16T23:23:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4318186496 bytes 2024-09-16T23:23:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4382731264 bytes 2024-09-16T23:23:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4390792192 bytes 2024-09-16T23:23:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4432355328 bytes 2024-09-16T23:23:39.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4527659008 bytes 2024-09-16T23:23:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4563764224 bytes 2024-09-16T23:23:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4620479488 bytes 2024-09-16T23:23:40.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4773922816 bytes 2024-09-16T23:23:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4791191552 bytes 2024-09-16T23:23:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4835847168 bytes 2024-09-16T23:23:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4879064064 bytes 2024-09-16T23:23:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4942554112 bytes 2024-09-16T23:23:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4988755968 bytes 2024-09-16T23:23:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 5071438848 bytes 2024-09-16T23:23:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 5114403840 bytes 2024-09-16T23:23:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 5164577792 bytes 2024-09-16T23:23:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_37 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:23:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-16T23:23:41.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_37': 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14406c15b1d2 2024-09-16T23:23:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14406c15b1d2 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:41 2024 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:41 2024 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:41 2024 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 33a3a258-5752-4a63-9f29-86e8aa61b524 2024-09-16T23:23:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=7 2024-09-16T23:23:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 36425728 bytes 2024-09-16T23:23:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 153759744 bytes 2024-09-16T23:23:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 167971840 bytes 2024-09-16T23:23:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 227868672 bytes 2024-09-16T23:23:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 311216128 bytes 2024-09-16T23:23:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 320305152 bytes 2024-09-16T23:23:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 400004096 bytes 2024-09-16T23:23:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 465222656 bytes 2024-09-16T23:23:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 487323648 bytes 2024-09-16T23:23:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 568538112 bytes 2024-09-16T23:23:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 599695360 bytes 2024-09-16T23:23:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 619587584 bytes 2024-09-16T23:23:43.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 635963392 bytes 2024-09-16T23:23:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 664175616 bytes 2024-09-16T23:23:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 712194048 bytes 2024-09-16T23:23:43.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 799737856 bytes 2024-09-16T23:23:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 818427904 bytes 2024-09-16T23:23:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 840478720 bytes 2024-09-16T23:23:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 879057920 bytes 2024-09-16T23:23:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 917986304 bytes 2024-09-16T23:23:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 936981504 bytes 2024-09-16T23:23:44.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 937559040 bytes 2024-09-16T23:23:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1028190208 bytes 2024-09-16T23:23:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1067807744 bytes 2024-09-16T23:23:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1098588160 bytes 2024-09-16T23:23:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1141338112 bytes 2024-09-16T23:23:45.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1207822336 bytes 2024-09-16T23:23:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1233145856 bytes 2024-09-16T23:23:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1271942144 bytes 2024-09-16T23:23:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1295946752 bytes 2024-09-16T23:23:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1303875584 bytes 2024-09-16T23:23:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1319686144 bytes 2024-09-16T23:23:45.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1372187648 bytes 2024-09-16T23:23:46.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1429389312 bytes 2024-09-16T23:23:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1442443264 bytes 2024-09-16T23:23:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1462381568 bytes 2024-09-16T23:23:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1495300096 bytes 2024-09-16T23:23:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1559397376 bytes 2024-09-16T23:23:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1589437440 bytes 2024-09-16T23:23:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1612048384 bytes 2024-09-16T23:23:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1702588416 bytes 2024-09-16T23:23:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1703239680 bytes 2024-09-16T23:23:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1703849984 bytes 2024-09-16T23:23:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1776486400 bytes 2024-09-16T23:23:47.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1807311872 bytes 2024-09-16T23:23:47.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1836782592 bytes 2024-09-16T23:23:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1955429376 bytes 2024-09-16T23:23:47.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1986319360 bytes 2024-09-16T23:23:47.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2016020480 bytes 2024-09-16T23:23:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2103844864 bytes 2024-09-16T23:23:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2123472896 bytes 2024-09-16T23:23:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2140705792 bytes 2024-09-16T23:23:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2194311168 bytes 2024-09-16T23:23:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2215367680 bytes 2024-09-16T23:23:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2282361856 bytes 2024-09-16T23:23:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:23:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_38 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:23:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:23:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-16T23:23:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:23:49.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-16T23:23:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_38': 2024-09-16T23:23:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 144cb0c67b9 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.144cb0c67b9 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:48 2024 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:48 2024 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:48 2024 2024-09-16T23:23:49.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 94476c47-d77a-4fe0-9be9-c62c51a92c96 2024-09-16T23:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:23:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:23:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:23:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:23:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:23:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 92907520 bytes 2024-09-16T23:23:50.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 161519616 bytes 2024-09-16T23:23:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 188081152 bytes 2024-09-16T23:23:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 242690048 bytes 2024-09-16T23:23:50.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 264383488 bytes 2024-09-16T23:23:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 281781248 bytes 2024-09-16T23:23:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 302474240 bytes 2024-09-16T23:23:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 309854208 bytes 2024-09-16T23:23:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 391208960 bytes 2024-09-16T23:23:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 448365568 bytes 2024-09-16T23:23:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 480314368 bytes 2024-09-16T23:23:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 496641024 bytes 2024-09-16T23:23:51.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 531510272 bytes 2024-09-16T23:23:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 563309568 bytes 2024-09-16T23:23:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 658365440 bytes 2024-09-16T23:23:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 677602304 bytes 2024-09-16T23:23:51.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 682883072 bytes 2024-09-16T23:23:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 726050816 bytes 2024-09-16T23:23:51.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 767442944 bytes 2024-09-16T23:23:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 856543232 bytes 2024-09-16T23:23:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 903073792 bytes 2024-09-16T23:23:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 954479616 bytes 2024-09-16T23:23:52.455 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9408864788001223) > weight (0.25) 2024-09-16T23:23:52.455 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.39968243023991) > weight (0.25) 2024-09-16T23:23:52.455 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8139081240722871) > weight (0.25) 2024-09-16T23:23:52.455 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T23:23:52.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T23:23:52.456 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T23:23:52.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:23:52.459+0000 7fee9fd18640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 36939) UID: 1000 2024-09-16T23:23:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1025721344 bytes 2024-09-16T23:23:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1079354368 bytes 2024-09-16T23:23:52.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1125763072 bytes 2024-09-16T23:23:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1126376448 bytes 2024-09-16T23:23:53.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1156129792 bytes 2024-09-16T23:23:53.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1168528384 bytes 2024-09-16T23:23:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1212065792 bytes 2024-09-16T23:23:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1237123072 bytes 2024-09-16T23:23:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1255323648 bytes 2024-09-16T23:23:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1268311040 bytes 2024-09-16T23:23:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1288582144 bytes 2024-09-16T23:23:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1319692288 bytes 2024-09-16T23:23:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1369542656 bytes 2024-09-16T23:23:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1391052800 bytes 2024-09-16T23:23:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1464332288 bytes 2024-09-16T23:23:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1502611456 bytes 2024-09-16T23:23:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1542053888 bytes 2024-09-16T23:23:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1564726272 bytes 2024-09-16T23:23:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1602574336 bytes 2024-09-16T23:23:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1680164864 bytes 2024-09-16T23:23:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1767303168 bytes 2024-09-16T23:23:55.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1810818048 bytes 2024-09-16T23:23:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1918077952 bytes 2024-09-16T23:23:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1927839744 bytes 2024-09-16T23:23:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1977577472 bytes 2024-09-16T23:23:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2011060224 bytes 2024-09-16T23:23:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2042782720 bytes 2024-09-16T23:23:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2102173696 bytes 2024-09-16T23:23:55.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2130131968 bytes 2024-09-16T23:23:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2172047360 bytes 2024-09-16T23:23:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2185101312 bytes 2024-09-16T23:23:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2217478144 bytes 2024-09-16T23:23:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2271700992 bytes 2024-09-16T23:23:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2326680576 bytes 2024-09-16T23:23:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2341215232 bytes 2024-09-16T23:23:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2387087360 bytes 2024-09-16T23:23:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2481545216 bytes 2024-09-16T23:23:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2502631424 bytes 2024-09-16T23:23:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2525481984 bytes 2024-09-16T23:23:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2599493632 bytes 2024-09-16T23:23:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2642526208 bytes 2024-09-16T23:23:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2699855872 bytes 2024-09-16T23:23:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2701989888 bytes 2024-09-16T23:23:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2752645120 bytes 2024-09-16T23:23:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2783403008 bytes 2024-09-16T23:23:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2786764800 bytes 2024-09-16T23:23:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2818642944 bytes 2024-09-16T23:23:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2856218624 bytes 2024-09-16T23:23:58.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2895729664 bytes 2024-09-16T23:23:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2953861120 bytes 2024-09-16T23:23:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2987405312 bytes 2024-09-16T23:23:58.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3059095552 bytes 2024-09-16T23:23:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3106273280 bytes 2024-09-16T23:23:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3154123776 bytes 2024-09-16T23:23:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3193748480 bytes 2024-09-16T23:23:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3231690752 bytes 2024-09-16T23:23:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3292899328 bytes 2024-09-16T23:23:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3327757312 bytes 2024-09-16T23:23:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3467503616 bytes 2024-09-16T23:23:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3524629504 bytes 2024-09-16T23:24:00.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3667130368 bytes 2024-09-16T23:24:00.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3711539200 bytes 2024-09-16T23:24:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3768208384 bytes 2024-09-16T23:24:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3768992768 bytes 2024-09-16T23:24:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3782622208 bytes 2024-09-16T23:24:00.457 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T23:24:00.457 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:24:00.457 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T23:24:00.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T23:24:00.458 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T23:24:00.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T23:24:00.461 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2024-09-16T23:24:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3814750208 bytes 2024-09-16T23:24:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3853272064 bytes 2024-09-16T23:24:00.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3879328768 bytes 2024-09-16T23:24:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3911411712 bytes 2024-09-16T23:24:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4003527680 bytes 2024-09-16T23:24:01.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4042744832 bytes 2024-09-16T23:24:01.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4088915968 bytes 2024-09-16T23:24:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4160545792 bytes 2024-09-16T23:24:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4184869888 bytes 2024-09-16T23:24:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_39 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-16T23:24:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-16T23:24:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_39': 2024-09-16T23:24:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:02.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 145d4a8fda8a 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.145d4a8fda8a 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:01 2024 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:01 2024 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:01 2024 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 25336153-2db6-45d0-90fa-87fcf4dfdbf9 2024-09-16T23:24:02.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:24:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:24:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:02.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 44864512 bytes 2024-09-16T23:24:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 54754304 bytes 2024-09-16T23:24:03.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 73310208 bytes 2024-09-16T23:24:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 151528448 bytes 2024-09-16T23:24:03.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 219292672 bytes 2024-09-16T23:24:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 247105536 bytes 2024-09-16T23:24:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 337997824 bytes 2024-09-16T23:24:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 399515648 bytes 2024-09-16T23:24:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 415864832 bytes 2024-09-16T23:24:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 440230912 bytes 2024-09-16T23:24:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 490227712 bytes 2024-09-16T23:24:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 509072384 bytes 2024-09-16T23:24:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 565141504 bytes 2024-09-16T23:24:04.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 605263872 bytes 2024-09-16T23:24:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 654281728 bytes 2024-09-16T23:24:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 696909824 bytes 2024-09-16T23:24:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 774322176 bytes 2024-09-16T23:24:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 791831552 bytes 2024-09-16T23:24:05.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 839788544 bytes 2024-09-16T23:24:05.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 898724864 bytes 2024-09-16T23:24:05.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 919205888 bytes 2024-09-16T23:24:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 942681088 bytes 2024-09-16T23:24:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1007865856 bytes 2024-09-16T23:24:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1008451584 bytes 2024-09-16T23:24:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1014467584 bytes 2024-09-16T23:24:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1062895616 bytes 2024-09-16T23:24:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1063489536 bytes 2024-09-16T23:24:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1084321792 bytes 2024-09-16T23:24:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1125054464 bytes 2024-09-16T23:24:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1148065792 bytes 2024-09-16T23:24:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1170860032 bytes 2024-09-16T23:24:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1218111488 bytes 2024-09-16T23:24:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1244328960 bytes 2024-09-16T23:24:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1293090816 bytes 2024-09-16T23:24:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1329277952 bytes 2024-09-16T23:24:07.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1356309504 bytes 2024-09-16T23:24:07.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1381612544 bytes 2024-09-16T23:24:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1469698048 bytes 2024-09-16T23:24:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1532646400 bytes 2024-09-16T23:24:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1554060288 bytes 2024-09-16T23:24:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1563384832 bytes 2024-09-16T23:24:07.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1608237056 bytes 2024-09-16T23:24:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1644304384 bytes 2024-09-16T23:24:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1659402240 bytes 2024-09-16T23:24:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1686842368 bytes 2024-09-16T23:24:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1731852288 bytes 2024-09-16T23:24:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1776311296 bytes 2024-09-16T23:24:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1783843840 bytes 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_40 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-16T23:24:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-16T23:24:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:09.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-16T23:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_40': 2024-09-16T23:24:09.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1469cbd97a8b 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1469cbd97a8b 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:08 2024 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:08 2024 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:08 2024 2024-09-16T23:24:09.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 709ec639-041d-469d-b611-2b988ea528b9 2024-09-16T23:24:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-16T23:24:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:24:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:24:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:09.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 46643200 bytes 2024-09-16T23:24:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 96112640 bytes 2024-09-16T23:24:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 134177792 bytes 2024-09-16T23:24:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 166788096 bytes 2024-09-16T23:24:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 252989440 bytes 2024-09-16T23:24:10.422 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:24:10.424+0000 7f821775c640 -1 reset not still connected to 0x55a21d760ea0 2024-09-16T23:24:10.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 305664000 bytes 2024-09-16T23:24:10.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 338936832 bytes 2024-09-16T23:24:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 339582976 bytes 2024-09-16T23:24:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 393059328 bytes 2024-09-16T23:24:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 430910464 bytes 2024-09-16T23:24:11.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 478809088 bytes 2024-09-16T23:24:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 501967872 bytes 2024-09-16T23:24:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 646764544 bytes 2024-09-16T23:24:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 732384256 bytes 2024-09-16T23:24:11.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 803641344 bytes 2024-09-16T23:24:11.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 840570880 bytes 2024-09-16T23:24:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 883731456 bytes 2024-09-16T23:24:12.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 906667008 bytes 2024-09-16T23:24:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 978400256 bytes 2024-09-16T23:24:12.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 989659136 bytes 2024-09-16T23:24:12.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1042288640 bytes 2024-09-16T23:24:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1078664192 bytes 2024-09-16T23:24:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1117465600 bytes 2024-09-16T23:24:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1138659328 bytes 2024-09-16T23:24:12.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1152068608 bytes 2024-09-16T23:24:13.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1188641792 bytes 2024-09-16T23:24:13.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1241287680 bytes 2024-09-16T23:24:13.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1311757312 bytes 2024-09-16T23:24:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1421274112 bytes 2024-09-16T23:24:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1450074112 bytes 2024-09-16T23:24:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1482050560 bytes 2024-09-16T23:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1551599616 bytes 2024-09-16T23:24:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1571842048 bytes 2024-09-16T23:24:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1606999040 bytes 2024-09-16T23:24:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1623325696 bytes 2024-09-16T23:24:14.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1643878400 bytes 2024-09-16T23:24:14.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1663490048 bytes 2024-09-16T23:24:14.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1731414016 bytes 2024-09-16T23:24:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1771256832 bytes 2024-09-16T23:24:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1811591168 bytes 2024-09-16T23:24:14.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1846351872 bytes 2024-09-16T23:24:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1884711936 bytes 2024-09-16T23:24:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1908497408 bytes 2024-09-16T23:24:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1936532480 bytes 2024-09-16T23:24:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1941468160 bytes 2024-09-16T23:24:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2009347072 bytes 2024-09-16T23:24:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2034919424 bytes 2024-09-16T23:24:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2067736576 bytes 2024-09-16T23:24:15.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2099254272 bytes 2024-09-16T23:24:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2121389056 bytes 2024-09-16T23:24:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2187745280 bytes 2024-09-16T23:24:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2202650624 bytes 2024-09-16T23:24:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2207853568 bytes 2024-09-16T23:24:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2243249152 bytes 2024-09-16T23:24:16.375 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:24:16.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:24:16.375 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T23:24:16.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:24:16.378+0000 7fd2d9466640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 45210) UID: 1000 2024-09-16T23:24:16.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2276745216 bytes 2024-09-16T23:24:16.527 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:24:16.529+0000 7fbcef505640 -1 reset not still connected to 0x56517f4fa680 2024-09-16T23:24:16.532 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:24:16.533+0000 7fbcef505640 -1 reset not still connected to 0x5651808dc270 2024-09-16T23:24:16.532 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:24:16.533+0000 7fbcef505640 -1 reset not still connected to 0x5651970a6f70 2024-09-16T23:24:16.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2280199168 bytes 2024-09-16T23:24:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2300083200 bytes 2024-09-16T23:24:16.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2347505664 bytes 2024-09-16T23:24:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2405533696 bytes 2024-09-16T23:24:17.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2446832640 bytes 2024-09-16T23:24:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2453993472 bytes 2024-09-16T23:24:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2482791424 bytes 2024-09-16T23:24:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2497755136 bytes 2024-09-16T23:24:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2546759680 bytes 2024-09-16T23:24:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2590054400 bytes 2024-09-16T23:24:17.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2613715968 bytes 2024-09-16T23:24:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2627650560 bytes 2024-09-16T23:24:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2654431232 bytes 2024-09-16T23:24:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2713104384 bytes 2024-09-16T23:24:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2735607808 bytes 2024-09-16T23:24:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2761496576 bytes 2024-09-16T23:24:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2782729216 bytes 2024-09-16T23:24:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2816610304 bytes 2024-09-16T23:24:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2840425472 bytes 2024-09-16T23:24:18.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2872188928 bytes 2024-09-16T23:24:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2958343168 bytes 2024-09-16T23:24:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3018686464 bytes 2024-09-16T23:24:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3055617024 bytes 2024-09-16T23:24:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3092222976 bytes 2024-09-16T23:24:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3173573632 bytes 2024-09-16T23:24:19.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3287425024 bytes 2024-09-16T23:24:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3337802752 bytes 2024-09-16T23:24:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_41 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2024-09-16T23:24:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:19.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:19.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-16T23:24:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-16T23:24:20.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_41': 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1474ab39f255 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1474ab39f255 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:19 2024 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:19 2024 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:19 2024 2024-09-16T23:24:20.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c513aa23-d6df-4ccf-ba45-b813bb34e2b8 2024-09-16T23:24:20.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-16T23:24:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:24:20.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=9 2024-09-16T23:24:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 52442112 bytes 2024-09-16T23:24:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 110774272 bytes 2024-09-16T23:24:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 133820416 bytes 2024-09-16T23:24:21.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 158286848 bytes 2024-09-16T23:24:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 193807360 bytes 2024-09-16T23:24:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 239169536 bytes 2024-09-16T23:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 239768576 bytes 2024-09-16T23:24:21.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 297263104 bytes 2024-09-16T23:24:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 361515008 bytes 2024-09-16T23:24:21.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 362122240 bytes 2024-09-16T23:24:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 434188288 bytes 2024-09-16T23:24:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 453611520 bytes 2024-09-16T23:24:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 491375616 bytes 2024-09-16T23:24:22.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 515949568 bytes 2024-09-16T23:24:22.376 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:24:22.376 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:24:22.376 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:24:22.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:24:22.377 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:24:22.379 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:24:22.380 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-16T23:24:22.532 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 585838592 bytes 2024-09-16T23:24:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 629020672 bytes 2024-09-16T23:24:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 650957824 bytes 2024-09-16T23:24:22.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 692301824 bytes 2024-09-16T23:24:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 708069376 bytes 2024-09-16T23:24:23.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 744108032 bytes 2024-09-16T23:24:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 796131328 bytes 2024-09-16T23:24:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 830027776 bytes 2024-09-16T23:24:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 858141696 bytes 2024-09-16T23:24:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 907348992 bytes 2024-09-16T23:24:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 915464192 bytes 2024-09-16T23:24:23.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 934865920 bytes 2024-09-16T23:24:23.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 954374144 bytes 2024-09-16T23:24:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1000379392 bytes 2024-09-16T23:24:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1062677504 bytes 2024-09-16T23:24:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1097964544 bytes 2024-09-16T23:24:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1122400256 bytes 2024-09-16T23:24:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1203598336 bytes 2024-09-16T23:24:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1240941568 bytes 2024-09-16T23:24:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1284295680 bytes 2024-09-16T23:24:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1309031424 bytes 2024-09-16T23:24:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1359410176 bytes 2024-09-16T23:24:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1375866880 bytes 2024-09-16T23:24:25.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1393762304 bytes 2024-09-16T23:24:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1439699968 bytes 2024-09-16T23:24:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1453625344 bytes 2024-09-16T23:24:25.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1513277440 bytes 2024-09-16T23:24:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1528868864 bytes 2024-09-16T23:24:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1580444672 bytes 2024-09-16T23:24:26.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1669625856 bytes 2024-09-16T23:24:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1688218624 bytes 2024-09-16T23:24:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1761470464 bytes 2024-09-16T23:24:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1829964800 bytes 2024-09-16T23:24:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1858725888 bytes 2024-09-16T23:24:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1935569920 bytes 2024-09-16T23:24:26.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1998257152 bytes 2024-09-16T23:24:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2016622592 bytes 2024-09-16T23:24:27.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2058579968 bytes 2024-09-16T23:24:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2077227008 bytes 2024-09-16T23:24:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2138928128 bytes 2024-09-16T23:24:27.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2179309568 bytes 2024-09-16T23:24:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2236896256 bytes 2024-09-16T23:24:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2312315904 bytes 2024-09-16T23:24:27.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2336104448 bytes 2024-09-16T23:24:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2341496832 bytes 2024-09-16T23:24:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2411634688 bytes 2024-09-16T23:24:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2430408704 bytes 2024-09-16T23:24:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2466265088 bytes 2024-09-16T23:24:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2549450752 bytes 2024-09-16T23:24:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2608613376 bytes 2024-09-16T23:24:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2629815296 bytes 2024-09-16T23:24:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2702724096 bytes 2024-09-16T23:24:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2774300672 bytes 2024-09-16T23:24:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2817193984 bytes 2024-09-16T23:24:29.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2881907712 bytes 2024-09-16T23:24:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2913373184 bytes 2024-09-16T23:24:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2927881216 bytes 2024-09-16T23:24:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_42 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-16T23:24:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-16T23:24:30.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:30.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-16T23:24:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_42': 2024-09-16T23:24:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 148934202071 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.148934202071 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:29 2024 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:29 2024 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:29 2024 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2a60c1ad-c75f-4e2d-a149-046ff2981c5f 2024-09-16T23:24:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:24:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:24:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 25439232 bytes 2024-09-16T23:24:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 141215744 bytes 2024-09-16T23:24:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 219668480 bytes 2024-09-16T23:24:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 280268800 bytes 2024-09-16T23:24:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 333664256 bytes 2024-09-16T23:24:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 372536320 bytes 2024-09-16T23:24:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 417770496 bytes 2024-09-16T23:24:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 456455168 bytes 2024-09-16T23:24:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 493239296 bytes 2024-09-16T23:24:32.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 535161856 bytes 2024-09-16T23:24:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 599377920 bytes 2024-09-16T23:24:32.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 622354432 bytes 2024-09-16T23:24:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 669984768 bytes 2024-09-16T23:24:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 724807680 bytes 2024-09-16T23:24:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 758852608 bytes 2024-09-16T23:24:32.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 879443968 bytes 2024-09-16T23:24:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 882142208 bytes 2024-09-16T23:24:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 914741248 bytes 2024-09-16T23:24:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 971292672 bytes 2024-09-16T23:24:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1021313024 bytes 2024-09-16T23:24:33.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1071825920 bytes 2024-09-16T23:24:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1133977600 bytes 2024-09-16T23:24:33.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1210468352 bytes 2024-09-16T23:24:33.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1272347648 bytes 2024-09-16T23:24:34.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1379162112 bytes 2024-09-16T23:24:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1405385728 bytes 2024-09-16T23:24:34.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1425149952 bytes 2024-09-16T23:24:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1483010048 bytes 2024-09-16T23:24:34.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1546118144 bytes 2024-09-16T23:24:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1573542912 bytes 2024-09-16T23:24:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1595237376 bytes 2024-09-16T23:24:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1628554240 bytes 2024-09-16T23:24:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1629164544 bytes 2024-09-16T23:24:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1696891904 bytes 2024-09-16T23:24:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1764909056 bytes 2024-09-16T23:24:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1787589632 bytes 2024-09-16T23:24:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1847459840 bytes 2024-09-16T23:24:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1937057792 bytes 2024-09-16T23:24:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1978819584 bytes 2024-09-16T23:24:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1979466752 bytes 2024-09-16T23:24:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1996738560 bytes 2024-09-16T23:24:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 2085570560 bytes 2024-09-16T23:24:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2152182784 bytes 2024-09-16T23:24:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2190999552 bytes 2024-09-16T23:24:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2236391424 bytes 2024-09-16T23:24:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2303796224 bytes 2024-09-16T23:24:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2366386176 bytes 2024-09-16T23:24:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2406020096 bytes 2024-09-16T23:24:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2407464960 bytes 2024-09-16T23:24:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2442661888 bytes 2024-09-16T23:24:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2520599552 bytes 2024-09-16T23:24:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2559337472 bytes 2024-09-16T23:24:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2647225344 bytes 2024-09-16T23:24:37.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2654879744 bytes 2024-09-16T23:24:37.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2677912576 bytes 2024-09-16T23:24:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2712941568 bytes 2024-09-16T23:24:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2766763008 bytes 2024-09-16T23:24:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2789758976 bytes 2024-09-16T23:24:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2857190400 bytes 2024-09-16T23:24:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2924883968 bytes 2024-09-16T23:24:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2990775296 bytes 2024-09-16T23:24:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 3074653184 bytes 2024-09-16T23:24:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 3138648064 bytes 2024-09-16T23:24:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3164094464 bytes 2024-09-16T23:24:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3225553920 bytes 2024-09-16T23:24:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3254062080 bytes 2024-09-16T23:24:39.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3332297728 bytes 2024-09-16T23:24:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3337320448 bytes 2024-09-16T23:24:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3360826368 bytes 2024-09-16T23:24:39.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3367035904 bytes 2024-09-16T23:24:40.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3459827712 bytes 2024-09-16T23:24:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3474611200 bytes 2024-09-16T23:24:40.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3555893248 bytes 2024-09-16T23:24:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3576480768 bytes 2024-09-16T23:24:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3597531136 bytes 2024-09-16T23:24:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3633786880 bytes 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_43 2024-09-16T23:24:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:24:40.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:40.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-16T23:24:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_43': 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14952565252d 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14952565252d 2024-09-16T23:24:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:40 2024 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:40 2024 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:40 2024 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: f0c279cd-dbe9-4652-a6a2-a4416fd6df30 2024-09-16T23:24:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:24:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:24:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:41.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 11739136 bytes 2024-09-16T23:24:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 54928384 bytes 2024-09-16T23:24:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 65705984 bytes 2024-09-16T23:24:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 110914560 bytes 2024-09-16T23:24:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 158454784 bytes 2024-09-16T23:24:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 182422528 bytes 2024-09-16T23:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 213629952 bytes 2024-09-16T23:24:42.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 243356672 bytes 2024-09-16T23:24:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 295859200 bytes 2024-09-16T23:24:42.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 333746176 bytes 2024-09-16T23:24:43.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 386866176 bytes 2024-09-16T23:24:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 393372672 bytes 2024-09-16T23:24:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 433729536 bytes 2024-09-16T23:24:43.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 537836544 bytes 2024-09-16T23:24:43.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 569541632 bytes 2024-09-16T23:24:43.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 570144768 bytes 2024-09-16T23:24:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 629156864 bytes 2024-09-16T23:24:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 666760192 bytes 2024-09-16T23:24:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 732984320 bytes 2024-09-16T23:24:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 807857152 bytes 2024-09-16T23:24:44.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 864114688 bytes 2024-09-16T23:24:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 905026560 bytes 2024-09-16T23:24:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 987663360 bytes 2024-09-16T23:24:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 993067008 bytes 2024-09-16T23:24:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1060953088 bytes 2024-09-16T23:24:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1086282752 bytes 2024-09-16T23:24:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1140009984 bytes 2024-09-16T23:24:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1176027136 bytes 2024-09-16T23:24:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1205568512 bytes 2024-09-16T23:24:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1242309632 bytes 2024-09-16T23:24:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1320646656 bytes 2024-09-16T23:24:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1397758976 bytes 2024-09-16T23:24:45.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1442539520 bytes 2024-09-16T23:24:46.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1479939072 bytes 2024-09-16T23:24:46.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1491920896 bytes 2024-09-16T23:24:46.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1573820416 bytes 2024-09-16T23:24:46.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1631969280 bytes 2024-09-16T23:24:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1653609472 bytes 2024-09-16T23:24:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1701381120 bytes 2024-09-16T23:24:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1742420992 bytes 2024-09-16T23:24:46.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1745090560 bytes 2024-09-16T23:24:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1778195456 bytes 2024-09-16T23:24:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1825699840 bytes 2024-09-16T23:24:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1861568512 bytes 2024-09-16T23:24:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1898290176 bytes 2024-09-16T23:24:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1928890368 bytes 2024-09-16T23:24:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1980300288 bytes 2024-09-16T23:24:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2031840256 bytes 2024-09-16T23:24:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2056933376 bytes 2024-09-16T23:24:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2061634560 bytes 2024-09-16T23:24:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2103248896 bytes 2024-09-16T23:24:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2128195584 bytes 2024-09-16T23:24:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2169403392 bytes 2024-09-16T23:24:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2246618112 bytes 2024-09-16T23:24:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2272280576 bytes 2024-09-16T23:24:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2387008512 bytes 2024-09-16T23:24:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2436789248 bytes 2024-09-16T23:24:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2457607168 bytes 2024-09-16T23:24:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2518619136 bytes 2024-09-16T23:24:49.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2539915264 bytes 2024-09-16T23:24:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2557375488 bytes 2024-09-16T23:24:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2587512832 bytes 2024-09-16T23:24:49.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2657651712 bytes 2024-09-16T23:24:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2674472960 bytes 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:24:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_44 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:24:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:24:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:24:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-16T23:24:50.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:24:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_44': 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14a17cee5a17 2024-09-16T23:24:50.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14a17cee5a17 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:49 2024 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:49 2024 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:49 2024 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 37e2b986-79ec-4fb7-9730-03a545b99092 2024-09-16T23:24:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:24:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:24:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-16T23:24:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:24:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:24:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:24:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:24:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:24:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:24:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:24:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 49952768 bytes 2024-09-16T23:24:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 109486080 bytes 2024-09-16T23:24:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 134758400 bytes 2024-09-16T23:24:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 158891008 bytes 2024-09-16T23:24:51.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 173698048 bytes 2024-09-16T23:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 243407872 bytes 2024-09-16T23:24:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 311352320 bytes 2024-09-16T23:24:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 354048000 bytes 2024-09-16T23:24:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 419428352 bytes 2024-09-16T23:24:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 500812800 bytes 2024-09-16T23:24:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 562203648 bytes 2024-09-16T23:24:52.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 569290752 bytes 2024-09-16T23:24:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 642824192 bytes 2024-09-16T23:24:52.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 703139840 bytes 2024-09-16T23:24:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 764687360 bytes 2024-09-16T23:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 839897088 bytes 2024-09-16T23:24:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 841483264 bytes 2024-09-16T23:24:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 899512320 bytes 2024-09-16T23:24:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 928443392 bytes 2024-09-16T23:24:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 985529344 bytes 2024-09-16T23:24:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1002372096 bytes 2024-09-16T23:24:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1045743616 bytes 2024-09-16T23:24:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1137944576 bytes 2024-09-16T23:24:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1162070016 bytes 2024-09-16T23:24:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1187314688 bytes 2024-09-16T23:24:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1217289216 bytes 2024-09-16T23:24:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1285084160 bytes 2024-09-16T23:24:54.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1310891008 bytes 2024-09-16T23:24:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1372129280 bytes 2024-09-16T23:24:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1439301632 bytes 2024-09-16T23:24:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1444515840 bytes 2024-09-16T23:24:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1469955072 bytes 2024-09-16T23:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1499318272 bytes 2024-09-16T23:24:55.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1529168896 bytes 2024-09-16T23:24:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1624839168 bytes 2024-09-16T23:24:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1723229184 bytes 2024-09-16T23:24:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1792797696 bytes 2024-09-16T23:24:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1867343872 bytes 2024-09-16T23:24:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1867976704 bytes 2024-09-16T23:24:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1923778560 bytes 2024-09-16T23:24:56.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1943584768 bytes 2024-09-16T23:24:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1959311360 bytes 2024-09-16T23:24:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2016444416 bytes 2024-09-16T23:24:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2052964352 bytes 2024-09-16T23:24:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2065445888 bytes 2024-09-16T23:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2090265600 bytes 2024-09-16T23:24:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2117582848 bytes 2024-09-16T23:24:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2179684352 bytes 2024-09-16T23:24:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2237601792 bytes 2024-09-16T23:24:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2281295872 bytes 2024-09-16T23:24:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2339213312 bytes 2024-09-16T23:24:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2354771968 bytes 2024-09-16T23:24:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2407388160 bytes 2024-09-16T23:24:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2437344256 bytes 2024-09-16T23:24:57.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2529339392 bytes 2024-09-16T23:24:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2630026240 bytes 2024-09-16T23:24:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2656892928 bytes 2024-09-16T23:24:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2723651584 bytes 2024-09-16T23:24:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2762052608 bytes 2024-09-16T23:24:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2789827584 bytes 2024-09-16T23:24:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2847845376 bytes 2024-09-16T23:24:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2932784128 bytes 2024-09-16T23:24:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2967204864 bytes 2024-09-16T23:24:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 3002078208 bytes 2024-09-16T23:24:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 3014556672 bytes 2024-09-16T23:24:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3080974336 bytes 2024-09-16T23:24:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3116963840 bytes 2024-09-16T23:24:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3165185024 bytes 2024-09-16T23:24:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3183816704 bytes 2024-09-16T23:24:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3229454336 bytes 2024-09-16T23:24:59.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3251790848 bytes 2024-09-16T23:24:59.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3280022528 bytes 2024-09-16T23:25:00.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3280845824 bytes 2024-09-16T23:25:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3321290752 bytes 2024-09-16T23:25:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3353922560 bytes 2024-09-16T23:25:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3421843456 bytes 2024-09-16T23:25:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3422489600 bytes 2024-09-16T23:25:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3472357376 bytes 2024-09-16T23:25:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3520314368 bytes 2024-09-16T23:25:00.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3563576320 bytes 2024-09-16T23:25:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3630973952 bytes 2024-09-16T23:25:01.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3634099200 bytes 2024-09-16T23:25:01.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3655658496 bytes 2024-09-16T23:25:01.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3694938112 bytes 2024-09-16T23:25:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3708498944 bytes 2024-09-16T23:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3740022784 bytes 2024-09-16T23:25:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3756713984 bytes 2024-09-16T23:25:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3850387456 bytes 2024-09-16T23:25:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3923196928 bytes 2024-09-16T23:25:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3987692544 bytes 2024-09-16T23:25:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4087203840 bytes 2024-09-16T23:25:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4154786816 bytes 2024-09-16T23:25:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4238760960 bytes 2024-09-16T23:25:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4267809792 bytes 2024-09-16T23:25:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4364849152 bytes 2024-09-16T23:25:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4426642432 bytes 2024-09-16T23:25:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4474558464 bytes 2024-09-16T23:25:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4496219136 bytes 2024-09-16T23:25:03.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4500293632 bytes 2024-09-16T23:25:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4551270400 bytes 2024-09-16T23:25:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4603688960 bytes 2024-09-16T23:25:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4639164416 bytes 2024-09-16T23:25:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_45 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-16T23:25:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-16T23:25:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-16T23:25:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_45': 2024-09-16T23:25:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:04.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14add7b3235b 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14add7b3235b 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:03 2024 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:03 2024 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:03 2024 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: a03530bd-5bbe-4ed9-9353-0fe8900fb0a8 2024-09-16T23:25:04.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:25:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=11 2024-09-16T23:25:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 78184448 bytes 2024-09-16T23:25:05.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 111384576 bytes 2024-09-16T23:25:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 194795520 bytes 2024-09-16T23:25:05.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 260324352 bytes 2024-09-16T23:25:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 354720768 bytes 2024-09-16T23:25:05.605 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:25:05.608+0000 7fbcef505640 -1 reset not still connected to 0x56517f4fa680 2024-09-16T23:25:05.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 386376704 bytes 2024-09-16T23:25:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 405115904 bytes 2024-09-16T23:25:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 498310144 bytes 2024-09-16T23:25:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 583955456 bytes 2024-09-16T23:25:06.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 615362560 bytes 2024-09-16T23:25:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 645549056 bytes 2024-09-16T23:25:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 687862784 bytes 2024-09-16T23:25:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 743375872 bytes 2024-09-16T23:25:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 757056512 bytes 2024-09-16T23:25:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 818283520 bytes 2024-09-16T23:25:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 870107136 bytes 2024-09-16T23:25:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 997696512 bytes 2024-09-16T23:25:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 1015813120 bytes 2024-09-16T23:25:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 1087017984 bytes 2024-09-16T23:25:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1134264320 bytes 2024-09-16T23:25:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1134879744 bytes 2024-09-16T23:25:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1153031168 bytes 2024-09-16T23:25:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1206926336 bytes 2024-09-16T23:25:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1215191040 bytes 2024-09-16T23:25:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1258621952 bytes 2024-09-16T23:25:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1281509376 bytes 2024-09-16T23:25:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1305825280 bytes 2024-09-16T23:25:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1340589056 bytes 2024-09-16T23:25:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1375676416 bytes 2024-09-16T23:25:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1440429056 bytes 2024-09-16T23:25:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1486345216 bytes 2024-09-16T23:25:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1561859072 bytes 2024-09-16T23:25:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1606829056 bytes 2024-09-16T23:25:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1648447488 bytes 2024-09-16T23:25:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1741034496 bytes 2024-09-16T23:25:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1781764096 bytes 2024-09-16T23:25:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1814455296 bytes 2024-09-16T23:25:09.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1828929536 bytes 2024-09-16T23:25:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1876143104 bytes 2024-09-16T23:25:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1930075136 bytes 2024-09-16T23:25:10.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1972267008 bytes 2024-09-16T23:25:10.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1985543168 bytes 2024-09-16T23:25:10.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2045818880 bytes 2024-09-16T23:25:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2081732608 bytes 2024-09-16T23:25:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2082338816 bytes 2024-09-16T23:25:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2100678656 bytes 2024-09-16T23:25:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2192144384 bytes 2024-09-16T23:25:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2247756800 bytes 2024-09-16T23:25:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2270774272 bytes 2024-09-16T23:25:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2289498112 bytes 2024-09-16T23:25:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2355441664 bytes 2024-09-16T23:25:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2444856320 bytes 2024-09-16T23:25:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2449236992 bytes 2024-09-16T23:25:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2449852416 bytes 2024-09-16T23:25:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2485007360 bytes 2024-09-16T23:25:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2555566080 bytes 2024-09-16T23:25:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2575931392 bytes 2024-09-16T23:25:12.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2622189568 bytes 2024-09-16T23:25:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2623788032 bytes 2024-09-16T23:25:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2655568896 bytes 2024-09-16T23:25:12.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2662583296 bytes 2024-09-16T23:25:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2687017984 bytes 2024-09-16T23:25:12.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2720735232 bytes 2024-09-16T23:25:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2771354624 bytes 2024-09-16T23:25:12.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2793158656 bytes 2024-09-16T23:25:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2863489024 bytes 2024-09-16T23:25:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2896253952 bytes 2024-09-16T23:25:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2918794240 bytes 2024-09-16T23:25:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2939659264 bytes 2024-09-16T23:25:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3006281728 bytes 2024-09-16T23:25:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3062525952 bytes 2024-09-16T23:25:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3074618368 bytes 2024-09-16T23:25:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3120878592 bytes 2024-09-16T23:25:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3207132160 bytes 2024-09-16T23:25:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3269947392 bytes 2024-09-16T23:25:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3288666112 bytes 2024-09-16T23:25:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3302851584 bytes 2024-09-16T23:25:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3357127680 bytes 2024-09-16T23:25:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3425269760 bytes 2024-09-16T23:25:14.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3546010624 bytes 2024-09-16T23:25:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3546632192 bytes 2024-09-16T23:25:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3615284224 bytes 2024-09-16T23:25:15.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3660012544 bytes 2024-09-16T23:25:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3734998016 bytes 2024-09-16T23:25:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3806267392 bytes 2024-09-16T23:25:15.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3875141632 bytes 2024-09-16T23:25:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3938481152 bytes 2024-09-16T23:25:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_46 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2024-09-16T23:25:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-16T23:25:16.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-16T23:25:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_46': 2024-09-16T23:25:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14b96d05b78e 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14b96d05b78e 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:15 2024 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:15 2024 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:15 2024 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ae447345-839d-464e-9a21-3546356877ae 2024-09-16T23:25:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:25:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:25:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 80969728 bytes 2024-09-16T23:25:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 160766976 bytes 2024-09-16T23:25:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 203680768 bytes 2024-09-16T23:25:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 242907136 bytes 2024-09-16T23:25:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 329063424 bytes 2024-09-16T23:25:17.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 388982784 bytes 2024-09-16T23:25:17.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 410830848 bytes 2024-09-16T23:25:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 439257088 bytes 2024-09-16T23:25:18.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 485650432 bytes 2024-09-16T23:25:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 486267904 bytes 2024-09-16T23:25:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 543188992 bytes 2024-09-16T23:25:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 623015936 bytes 2024-09-16T23:25:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 682375168 bytes 2024-09-16T23:25:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 755729408 bytes 2024-09-16T23:25:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 847638528 bytes 2024-09-16T23:25:18.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 865477632 bytes 2024-09-16T23:25:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 930950144 bytes 2024-09-16T23:25:19.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 988168192 bytes 2024-09-16T23:25:19.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 1031309312 bytes 2024-09-16T23:25:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1101230080 bytes 2024-09-16T23:25:19.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1150947328 bytes 2024-09-16T23:25:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1179281408 bytes 2024-09-16T23:25:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1211783168 bytes 2024-09-16T23:25:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1301107712 bytes 2024-09-16T23:25:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1384876032 bytes 2024-09-16T23:25:20.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1408047104 bytes 2024-09-16T23:25:20.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1462324224 bytes 2024-09-16T23:25:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1498169344 bytes 2024-09-16T23:25:20.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1526873088 bytes 2024-09-16T23:25:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1568028672 bytes 2024-09-16T23:25:20.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1610772480 bytes 2024-09-16T23:25:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1611385856 bytes 2024-09-16T23:25:21.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1612010496 bytes 2024-09-16T23:25:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1653116928 bytes 2024-09-16T23:25:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1654195200 bytes 2024-09-16T23:25:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1654814720 bytes 2024-09-16T23:25:21.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1683562496 bytes 2024-09-16T23:25:21.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1700376576 bytes 2024-09-16T23:25:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1742200832 bytes 2024-09-16T23:25:21.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1762109440 bytes 2024-09-16T23:25:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1832982528 bytes 2024-09-16T23:25:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1867245568 bytes 2024-09-16T23:25:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1901355008 bytes 2024-09-16T23:25:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1922714624 bytes 2024-09-16T23:25:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1933102080 bytes 2024-09-16T23:25:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1998823424 bytes 2024-09-16T23:25:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2020344832 bytes 2024-09-16T23:25:22.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2086377472 bytes 2024-09-16T23:25:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2106692608 bytes 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_47 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2024-09-16T23:25:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:22.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-16T23:25:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_47': 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14c56955718 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14c56955718 2024-09-16T23:25:23.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:22 2024 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:22 2024 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:22 2024 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 52b8a006-6595-4b78-a489-5a8e09247cf9 2024-09-16T23:25:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:25:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:25:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:23.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 22193152 bytes 2024-09-16T23:25:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 50070528 bytes 2024-09-16T23:25:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 71832576 bytes 2024-09-16T23:25:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 113488896 bytes 2024-09-16T23:25:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 170223616 bytes 2024-09-16T23:25:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 213663744 bytes 2024-09-16T23:25:24.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 238219264 bytes 2024-09-16T23:25:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 277844992 bytes 2024-09-16T23:25:24.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 363416576 bytes 2024-09-16T23:25:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 412103680 bytes 2024-09-16T23:25:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 453281792 bytes 2024-09-16T23:25:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 488698880 bytes 2024-09-16T23:25:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 500052992 bytes 2024-09-16T23:25:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 550351872 bytes 2024-09-16T23:25:25.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 570628096 bytes 2024-09-16T23:25:25.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 580730880 bytes 2024-09-16T23:25:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 650304512 bytes 2024-09-16T23:25:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 669388800 bytes 2024-09-16T23:25:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 687790080 bytes 2024-09-16T23:25:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 826145792 bytes 2024-09-16T23:25:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 897185792 bytes 2024-09-16T23:25:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 958269440 bytes 2024-09-16T23:25:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 999327744 bytes 2024-09-16T23:25:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1019928576 bytes 2024-09-16T23:25:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1035242496 bytes 2024-09-16T23:25:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1073596416 bytes 2024-09-16T23:25:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1098565632 bytes 2024-09-16T23:25:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1161842688 bytes 2024-09-16T23:25:27.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1197101056 bytes 2024-09-16T23:25:27.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1218392064 bytes 2024-09-16T23:25:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1282781184 bytes 2024-09-16T23:25:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1314751488 bytes 2024-09-16T23:25:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1319186432 bytes 2024-09-16T23:25:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1347720192 bytes 2024-09-16T23:25:27.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1387008000 bytes 2024-09-16T23:25:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1437414400 bytes 2024-09-16T23:25:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1532604416 bytes 2024-09-16T23:25:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1586942976 bytes 2024-09-16T23:25:28.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1639390208 bytes 2024-09-16T23:25:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1687500800 bytes 2024-09-16T23:25:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1748820992 bytes 2024-09-16T23:25:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1780263936 bytes 2024-09-16T23:25:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1860477952 bytes 2024-09-16T23:25:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1873605632 bytes 2024-09-16T23:25:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1984318464 bytes 2024-09-16T23:25:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1996527616 bytes 2024-09-16T23:25:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2050968576 bytes 2024-09-16T23:25:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2093383680 bytes 2024-09-16T23:25:29.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2105791488 bytes 2024-09-16T23:25:29.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2119997440 bytes 2024-09-16T23:25:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2142668800 bytes 2024-09-16T23:25:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2170242048 bytes 2024-09-16T23:25:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2170850304 bytes 2024-09-16T23:25:30.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2187273216 bytes 2024-09-16T23:25:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2212595712 bytes 2024-09-16T23:25:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2248757248 bytes 2024-09-16T23:25:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2307799040 bytes 2024-09-16T23:25:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2376457216 bytes 2024-09-16T23:25:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2392973312 bytes 2024-09-16T23:25:30.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2409904128 bytes 2024-09-16T23:25:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2460598272 bytes 2024-09-16T23:25:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2508349440 bytes 2024-09-16T23:25:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2533741568 bytes 2024-09-16T23:25:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2609895424 bytes 2024-09-16T23:25:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2637778944 bytes 2024-09-16T23:25:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2675850240 bytes 2024-09-16T23:25:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2758225920 bytes 2024-09-16T23:25:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_48 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:31.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:31.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2024-09-16T23:25:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:31.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-16T23:25:32.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-16T23:25:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_48': 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14d1794a832e 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14d1794a832e 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:31 2024 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:31 2024 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:31 2024 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:32.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 31266f8d-efc2-46a5-bb23-092b5e031f3a 2024-09-16T23:25:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-16T23:25:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:25:32.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=8 2024-09-16T23:25:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:32.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:33.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 34312192 bytes 2024-09-16T23:25:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 125921280 bytes 2024-09-16T23:25:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 143190016 bytes 2024-09-16T23:25:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 210376704 bytes 2024-09-16T23:25:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 299863040 bytes 2024-09-16T23:25:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 335537152 bytes 2024-09-16T23:25:33.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 369035264 bytes 2024-09-16T23:25:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 383661056 bytes 2024-09-16T23:25:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 403534848 bytes 2024-09-16T23:25:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 430542848 bytes 2024-09-16T23:25:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 452780032 bytes 2024-09-16T23:25:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 525556736 bytes 2024-09-16T23:25:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 580832256 bytes 2024-09-16T23:25:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 635097088 bytes 2024-09-16T23:25:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 716191744 bytes 2024-09-16T23:25:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 781810688 bytes 2024-09-16T23:25:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 835484672 bytes 2024-09-16T23:25:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 861727744 bytes 2024-09-16T23:25:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 886109184 bytes 2024-09-16T23:25:35.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 909126656 bytes 2024-09-16T23:25:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 946745344 bytes 2024-09-16T23:25:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 972610560 bytes 2024-09-16T23:25:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 991777792 bytes 2024-09-16T23:25:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1027985408 bytes 2024-09-16T23:25:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1034335232 bytes 2024-09-16T23:25:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1034941440 bytes 2024-09-16T23:25:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1064375296 bytes 2024-09-16T23:25:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1067559936 bytes 2024-09-16T23:25:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1112378368 bytes 2024-09-16T23:25:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1162130432 bytes 2024-09-16T23:25:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1183033344 bytes 2024-09-16T23:25:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1217284096 bytes 2024-09-16T23:25:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1263891456 bytes 2024-09-16T23:25:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1285908480 bytes 2024-09-16T23:25:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1314670592 bytes 2024-09-16T23:25:37.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1338333184 bytes 2024-09-16T23:25:37.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1381412864 bytes 2024-09-16T23:25:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1472948224 bytes 2024-09-16T23:25:37.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1521224704 bytes 2024-09-16T23:25:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1542929408 bytes 2024-09-16T23:25:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1590925312 bytes 2024-09-16T23:25:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1627995136 bytes 2024-09-16T23:25:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1695211520 bytes 2024-09-16T23:25:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1727746048 bytes 2024-09-16T23:25:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1765163008 bytes 2024-09-16T23:25:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1799554048 bytes 2024-09-16T23:25:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1842993152 bytes 2024-09-16T23:25:38.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1850337280 bytes 2024-09-16T23:25:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1909239808 bytes 2024-09-16T23:25:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1984160768 bytes 2024-09-16T23:25:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1995052032 bytes 2024-09-16T23:25:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2098259968 bytes 2024-09-16T23:25:39.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2142109696 bytes 2024-09-16T23:25:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2166001664 bytes 2024-09-16T23:25:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2179161088 bytes 2024-09-16T23:25:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2233235456 bytes 2024-09-16T23:25:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2263086080 bytes 2024-09-16T23:25:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2323872768 bytes 2024-09-16T23:25:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2369007616 bytes 2024-09-16T23:25:39.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2460860416 bytes 2024-09-16T23:25:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2563056640 bytes 2024-09-16T23:25:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2624983040 bytes 2024-09-16T23:25:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2667361280 bytes 2024-09-16T23:25:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2711286784 bytes 2024-09-16T23:25:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2711915520 bytes 2024-09-16T23:25:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2721540096 bytes 2024-09-16T23:25:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2745757696 bytes 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_49 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2024-09-16T23:25:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-16T23:25:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-16T23:25:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_49': 2024-09-16T23:25:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14dd64dc46db 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14dd64dc46db 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:40 2024 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:40 2024 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:40 2024 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:41.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: aaa3557b-c9d4-42bc-827a-961b199be7de 2024-09-16T23:25:41.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:25:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:25:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 21400576 bytes 2024-09-16T23:25:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 90373120 bytes 2024-09-16T23:25:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 97829888 bytes 2024-09-16T23:25:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 167719936 bytes 2024-09-16T23:25:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 205615104 bytes 2024-09-16T23:25:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 260283392 bytes 2024-09-16T23:25:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 327511040 bytes 2024-09-16T23:25:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 394287104 bytes 2024-09-16T23:25:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 427761664 bytes 2024-09-16T23:25:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 478347264 bytes 2024-09-16T23:25:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 520814592 bytes 2024-09-16T23:25:43.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 640818176 bytes 2024-09-16T23:25:43.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 684428288 bytes 2024-09-16T23:25:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 822445056 bytes 2024-09-16T23:25:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 848727040 bytes 2024-09-16T23:25:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 853038080 bytes 2024-09-16T23:25:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 901743616 bytes 2024-09-16T23:25:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 936290304 bytes 2024-09-16T23:25:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 967690240 bytes 2024-09-16T23:25:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 1022898176 bytes 2024-09-16T23:25:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 1049409536 bytes 2024-09-16T23:25:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1105810432 bytes 2024-09-16T23:25:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1131527168 bytes 2024-09-16T23:25:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1171230720 bytes 2024-09-16T23:25:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1265680384 bytes 2024-09-16T23:25:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1292538880 bytes 2024-09-16T23:25:45.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1337944064 bytes 2024-09-16T23:25:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1378573312 bytes 2024-09-16T23:25:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1411599360 bytes 2024-09-16T23:25:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1456630784 bytes 2024-09-16T23:25:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1480795136 bytes 2024-09-16T23:25:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1509808128 bytes 2024-09-16T23:25:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1555021824 bytes 2024-09-16T23:25:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1601860608 bytes 2024-09-16T23:25:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1635461120 bytes 2024-09-16T23:25:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1678418944 bytes 2024-09-16T23:25:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1719277568 bytes 2024-09-16T23:25:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1743941632 bytes 2024-09-16T23:25:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1769242624 bytes 2024-09-16T23:25:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1824165888 bytes 2024-09-16T23:25:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1824782336 bytes 2024-09-16T23:25:47.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1884462080 bytes 2024-09-16T23:25:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1910802432 bytes 2024-09-16T23:25:47.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1930394624 bytes 2024-09-16T23:25:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1986455552 bytes 2024-09-16T23:25:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2021545984 bytes 2024-09-16T23:25:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2061645824 bytes 2024-09-16T23:25:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2066197504 bytes 2024-09-16T23:25:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2096791552 bytes 2024-09-16T23:25:47.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2104401920 bytes 2024-09-16T23:25:48.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2105026560 bytes 2024-09-16T23:25:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2158281728 bytes 2024-09-16T23:25:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2196551680 bytes 2024-09-16T23:25:48.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2215877632 bytes 2024-09-16T23:25:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2262739968 bytes 2024-09-16T23:25:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2283244544 bytes 2024-09-16T23:25:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2377314304 bytes 2024-09-16T23:25:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2429169664 bytes 2024-09-16T23:25:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2454531072 bytes 2024-09-16T23:25:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2525053952 bytes 2024-09-16T23:25:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2609096704 bytes 2024-09-16T23:25:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2718106624 bytes 2024-09-16T23:25:49.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2751814656 bytes 2024-09-16T23:25:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2793172992 bytes 2024-09-16T23:25:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2825683968 bytes 2024-09-16T23:25:49.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2848568320 bytes 2024-09-16T23:25:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2909218816 bytes 2024-09-16T23:25:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2964670464 bytes 2024-09-16T23:25:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2965298176 bytes 2024-09-16T23:25:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3007286272 bytes 2024-09-16T23:25:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3061440512 bytes 2024-09-16T23:25:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3150200832 bytes 2024-09-16T23:25:50.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3180284928 bytes 2024-09-16T23:25:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3226312704 bytes 2024-09-16T23:25:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3244214272 bytes 2024-09-16T23:25:51.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3270674432 bytes 2024-09-16T23:25:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3294956544 bytes 2024-09-16T23:25:51.380 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3352864750455352) > weight (0.25) 2024-09-16T23:25:51.380 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T23:25:51.381 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T23:25:51.381 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T23:25:51.384 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:25:51.383+0000 7fe4807ef640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 46980) UID: 1000 2024-09-16T23:25:51.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3298275328 bytes 2024-09-16T23:25:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3337078784 bytes 2024-09-16T23:25:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3405352960 bytes 2024-09-16T23:25:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3457561600 bytes 2024-09-16T23:25:51.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3458476032 bytes 2024-09-16T23:25:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3500429312 bytes 2024-09-16T23:25:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3522545664 bytes 2024-09-16T23:25:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3592260608 bytes 2024-09-16T23:25:52.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3625797632 bytes 2024-09-16T23:25:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3638102016 bytes 2024-09-16T23:25:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3715663872 bytes 2024-09-16T23:25:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3744061440 bytes 2024-09-16T23:25:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3787449344 bytes 2024-09-16T23:25:52.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3851823104 bytes 2024-09-16T23:25:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3856983040 bytes 2024-09-16T23:25:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3868822528 bytes 2024-09-16T23:25:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3932781568 bytes 2024-09-16T23:25:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3960060928 bytes 2024-09-16T23:25:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3981900800 bytes 2024-09-16T23:25:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4104999936 bytes 2024-09-16T23:25:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4124424192 bytes 2024-09-16T23:25:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4142962688 bytes 2024-09-16T23:25:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4177829888 bytes 2024-09-16T23:25:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4253006848 bytes 2024-09-16T23:25:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4279794688 bytes 2024-09-16T23:25:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4285665280 bytes 2024-09-16T23:25:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4305758208 bytes 2024-09-16T23:25:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4333196288 bytes 2024-09-16T23:25:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4333810688 bytes 2024-09-16T23:25:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4361620480 bytes 2024-09-16T23:25:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4386173952 bytes 2024-09-16T23:25:54.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_50 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n snapshot ']' 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 128M ']' 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=128M 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:25:54.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ is_pool_mirror_mode_image mirror 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' image = image ']' 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:25:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:25:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-16T23:25:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:25:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-16T23:25:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_50': 2024-09-16T23:25:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:25:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 14e9dfbdaff6 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.14e9dfbdaff6 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:54 2024 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:54 2024 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:54 2024 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:25:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b056f8d2-1f2f-4deb-8a45-c78bb6111629 2024-09-16T23:25:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n '' ']' 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:25:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:25:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=14 2024-09-16T23:25:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:25:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:25:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 54396928 bytes 2024-09-16T23:25:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 106688512 bytes 2024-09-16T23:25:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 228154368 bytes 2024-09-16T23:25:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 248776704 bytes 2024-09-16T23:25:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 281252864 bytes 2024-09-16T23:25:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 321204224 bytes 2024-09-16T23:25:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 358270976 bytes 2024-09-16T23:25:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 367779840 bytes 2024-09-16T23:25:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 406604800 bytes 2024-09-16T23:25:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 443404288 bytes 2024-09-16T23:25:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 472713216 bytes 2024-09-16T23:25:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 480054272 bytes 2024-09-16T23:25:57.382 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T23:25:57.382 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:25:57.382 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T23:25:57.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T23:25:57.383 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T23:25:57.387 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T23:25:57.387 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2024-09-16T23:25:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 530854912 bytes 2024-09-16T23:25:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 579116032 bytes 2024-09-16T23:25:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 618170368 bytes 2024-09-16T23:25:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 698409984 bytes 2024-09-16T23:25:58.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 799922176 bytes 2024-09-16T23:25:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 890568704 bytes 2024-09-16T23:25:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 951925760 bytes 2024-09-16T23:25:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 969888768 bytes 2024-09-16T23:25:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 990869504 bytes 2024-09-16T23:25:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1013756928 bytes 2024-09-16T23:25:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1025819648 bytes 2024-09-16T23:25:58.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1097287680 bytes 2024-09-16T23:25:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1132417024 bytes 2024-09-16T23:25:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1181315072 bytes 2024-09-16T23:25:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1224055808 bytes 2024-09-16T23:25:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1266822144 bytes 2024-09-16T23:25:59.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6307103278144734) > weight (0.25) 2024-09-16T23:25:59.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6335604630304504) > weight (0.25) 2024-09-16T23:25:59.462 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.60480938570624) > weight (0.25) 2024-09-16T23:25:59.462 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T23:25:59.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T23:25:59.463 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-16T23:25:59.463 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T23:25:59.463 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:25:59.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:25:59.465+0000 7f18b1dd7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 48765) UID: 1000 2024-09-16T23:25:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1332028416 bytes 2024-09-16T23:25:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1386229760 bytes 2024-09-16T23:25:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1399427072 bytes 2024-09-16T23:25:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1437053952 bytes 2024-09-16T23:25:59.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1475439616 bytes 2024-09-16T23:26:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1496814592 bytes 2024-09-16T23:26:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1530015744 bytes 2024-09-16T23:26:00.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1547744256 bytes 2024-09-16T23:26:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1665353728 bytes 2024-09-16T23:26:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1796263936 bytes 2024-09-16T23:26:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1833408512 bytes 2024-09-16T23:26:01.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1880473600 bytes 2024-09-16T23:26:01.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1932644352 bytes 2024-09-16T23:26:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1955667968 bytes 2024-09-16T23:26:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1999141888 bytes 2024-09-16T23:26:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2009267200 bytes 2024-09-16T23:26:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2045770752 bytes 2024-09-16T23:26:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2128054272 bytes 2024-09-16T23:26:01.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2163782656 bytes 2024-09-16T23:26:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2279932928 bytes 2024-09-16T23:26:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2357700608 bytes 2024-09-16T23:26:02.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2362713088 bytes 2024-09-16T23:26:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2408912896 bytes 2024-09-16T23:26:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2419131392 bytes 2024-09-16T23:26:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2493580288 bytes 2024-09-16T23:26:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2534233088 bytes 2024-09-16T23:26:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2575491072 bytes 2024-09-16T23:26:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2588763136 bytes 2024-09-16T23:26:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2616292352 bytes 2024-09-16T23:26:03.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2654196736 bytes 2024-09-16T23:26:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2654799872 bytes 2024-09-16T23:26:03.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2679571456 bytes 2024-09-16T23:26:03.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2742103040 bytes 2024-09-16T23:26:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2784872448 bytes 2024-09-16T23:26:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2785502208 bytes 2024-09-16T23:26:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2816014336 bytes 2024-09-16T23:26:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2868478976 bytes 2024-09-16T23:26:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2952238080 bytes 2024-09-16T23:26:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2976070656 bytes 2024-09-16T23:26:04.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2991017984 bytes 2024-09-16T23:26:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3029561344 bytes 2024-09-16T23:26:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3041295360 bytes 2024-09-16T23:26:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3097986048 bytes 2024-09-16T23:26:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3133210624 bytes 2024-09-16T23:26:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3170969600 bytes 2024-09-16T23:26:05.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3237494784 bytes 2024-09-16T23:26:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3270853632 bytes 2024-09-16T23:26:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3328003072 bytes 2024-09-16T23:26:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3395898368 bytes 2024-09-16T23:26:05.565 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T23:26:05.565 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T23:26:05.565 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T23:26:05.569 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T23:26:05.569 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-16T23:26:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3421350912 bytes 2024-09-16T23:26:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3454970880 bytes 2024-09-16T23:26:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3472264192 bytes 2024-09-16T23:26:06.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3532991488 bytes 2024-09-16T23:26:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3571789824 bytes 2024-09-16T23:26:06.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3629280256 bytes 2024-09-16T23:26:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3634892800 bytes 2024-09-16T23:26:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3680218112 bytes 2024-09-16T23:26:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3707854848 bytes 2024-09-16T23:26:06.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3721468928 bytes 2024-09-16T23:26:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3763177472 bytes 2024-09-16T23:26:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3799641088 bytes 2024-09-16T23:26:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3893105664 bytes 2024-09-16T23:26:07.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3909795840 bytes 2024-09-16T23:26:07.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4006050816 bytes 2024-09-16T23:26:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 4007860224 bytes 2024-09-16T23:26:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 4039433216 bytes 2024-09-16T23:26:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4040047616 bytes 2024-09-16T23:26:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4078505984 bytes 2024-09-16T23:26:07.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4103678976 bytes 2024-09-16T23:26:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4140686336 bytes 2024-09-16T23:26:08.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4143415296 bytes 2024-09-16T23:26:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4214682624 bytes 2024-09-16T23:26:08.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4224708608 bytes 2024-09-16T23:26:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4227574784 bytes 2024-09-16T23:26:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4279455744 bytes 2024-09-16T23:26:08.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4324193280 bytes 2024-09-16T23:26:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4362098688 bytes 2024-09-16T23:26:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4428058624 bytes 2024-09-16T23:26:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4499878912 bytes 2024-09-16T23:26:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4544645120 bytes 2024-09-16T23:26:09.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4545230848 bytes 2024-09-16T23:26:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4586650624 bytes 2024-09-16T23:26:09.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4661964800 bytes 2024-09-16T23:26:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4782135296 bytes 2024-09-16T23:26:09.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4823156736 bytes 2024-09-16T23:26:09.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11400 IOs, 4859998208 bytes 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_count=50 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local count 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_count=0 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:26:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:26:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:26:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T23:26:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'images: ' 2024-09-16T23:26:09.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 2 2024-09-16T23:26:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ count=50 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 50 = 50 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state 2024-09-16T23:26:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:26:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:26:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:26:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-16T23:26:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'image health:' 2024-09-16T23:26:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 3 2024-09-16T23:26:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=OK 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = ERROR 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = OK 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_count=50 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local count 2024-09-16T23:26:09.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_count=0 2024-09-16T23:26:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:26:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:26:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:26:09.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T23:26:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'images: ' 2024-09-16T23:26:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 2 2024-09-16T23:26:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:26:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ count=50 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 50 = 50 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state 2024-09-16T23:26:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T23:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'image health:' 2024-09-16T23:26:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 3 2024-09-16T23:26:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=WARNING 2024-09-16T23:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test WARNING = ERROR 2024-09-16T23:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test WARNING = OK 2024-09-16T23:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:26:10.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 2 -ne 1 2024-09-16T23:26:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 30 2024-09-16T23:26:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T23:26:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'image health:' 2024-09-16T23:26:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 3 2024-09-16T23:26:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=OK 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = ERROR 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = OK 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: compare many images' 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:26:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:26:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:26:40 TEST: compare many images 2024-09-16T23:26:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:26:40 TEST: compare many images 2024-09-16T23:26:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 50 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_1 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:40.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:26:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:26:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:26:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:26:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:26:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:26:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:26:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:26:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:26:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:26:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:26:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-16T23:26:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_1'] 2024-09-16T23:26:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:26:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:26:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:26:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:26:42.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:26:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:42.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2024-09-16T23:26:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:26:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:26:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:26:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:26:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:26:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:26:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:26:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-16T23:26:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_1.status 2024-09-16T23:26:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.status 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_1.status 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-16T23:26:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 86 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.status 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.status 2024-09-16T23:26:43.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-16T23:26:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:26:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=86 2024-09-16T23:26:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:43.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T23:26:43.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=35 2024-09-16T23:26:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 86 = 35 2024-09-16T23:26:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:26:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T23:26:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:26:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:26:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:26:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:26:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 86 = '' 2024-09-16T23:26:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:26:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T23:26:44.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:26:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:26:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:26:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:26:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 86 = '' 2024-09-16T23:26:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:44.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:26:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.status 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-16T23:26:46.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=86 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 86 = 86 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:26:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:26:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-16T23:26:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_1': 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c97d8fb50c 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c97d8fb50c 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:34 2024 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:34 2024 2024-09-16T23:26:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:34 2024 2024-09-16T23:26:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:26:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:26:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:26:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2fda3e9f-ae26-4e8f-b0fd-5a14428de02e 2024-09-16T23:26:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:26:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-16T23:26:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.export 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.export 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.export 2024-09-16T23:26:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.export 2024-09-16T23:26:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:26:47.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.export 2024-09-16T23:26:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:26:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.export 2024-09-16T23:26:47.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_1.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_1.export 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_2 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:48.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:48.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:26:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:26:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:26:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:26:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:26:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:26:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-16T23:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_2'] 2024-09-16T23:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:26:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:26:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:26:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:26:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-16T23:26:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:26:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:26:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:26:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:26:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_2 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_2 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_2.status 2024-09-16T23:26:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:49.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.status 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_2.status 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-16T23:26:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 88 2024-09-16T23:26:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.status 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.status 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-16T23:26:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:50.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:26:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=88 2024-09-16T23:26:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:26:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T23:26:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=36 2024-09-16T23:26:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 88 = 36 2024-09-16T23:26:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T23:26:50.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:26:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:26:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:26:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:26:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 88 = '' 2024-09-16T23:26:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:26:51.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.status 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-16T23:26:51.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:51.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=88 2024-09-16T23:26:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 88 = 88 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:26:51.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:26:52.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-16T23:26:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_2': 2024-09-16T23:26:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:26:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c99e960849 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c99e960849 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:43 2024 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:43 2024 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:43 2024 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:26:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:26:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 72e6208d-3742-4bfa-8918-249d11759c7e 2024-09-16T23:26:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.export 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.export 2024-09-16T23:26:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.export 2024-09-16T23:26:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.export 2024-09-16T23:26:52.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:26:52.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.export 2024-09-16T23:26:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:26:53.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.export 2024-09-16T23:26:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_2.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_2.export 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_3 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:26:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:26:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-16T23:26:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:26:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:26:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:26:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:26:55.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:26:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:26:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:26:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:26:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-16T23:26:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_3'] 2024-09-16T23:26:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:26:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:26:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:26:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:26:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:26:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:26:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:26:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:26:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:26:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-16T23:26:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:26:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:26:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:26:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:26:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:26:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:26:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:26:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_3 2024-09-16T23:26:55.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_3 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_3.status 2024-09-16T23:26:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.status 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_3.status 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-16T23:26:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 90 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.status 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.status 2024-09-16T23:26:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-16T23:26:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=90 2024-09-16T23:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:26:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T23:26:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=37 2024-09-16T23:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 90 = 37 2024-09-16T23:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:26:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T23:26:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:56.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 90 = '' 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:26:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T23:26:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:26:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:26:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:26:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:26:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:26:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 90 = '' 2024-09-16T23:26:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:26:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.status 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-16T23:26:59.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:26:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:26:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=90 2024-09-16T23:26:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 90 = 90 2024-09-16T23:26:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:26:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-16T23:27:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_3': 2024-09-16T23:27:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9ad27f3f6 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9ad27f3f6 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:17:57 2024 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:17:57 2024 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:17:57 2024 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 7dc49f0a-1a35-4bab-aea4-16de6b2f77bc 2024-09-16T23:27:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-16T23:27:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:27:00.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.export 2024-09-16T23:27:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.export 2024-09-16T23:27:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.export 2024-09-16T23:27:00.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.export 2024-09-16T23:27:00.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:00.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.export 2024-09-16T23:27:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.export 2024-09-16T23:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_3.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_3.export 2024-09-16T23:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_4 2024-09-16T23:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-16T23:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-16T23:27:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-16T23:27:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:02.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-16T23:27:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2024-09-16T23:27:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:02.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:02.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-16T23:27:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_4 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_4 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_4.status 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.status 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-16T23:27:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_4.status 2024-09-16T23:27:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-16T23:27:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-16T23:27:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 92 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.status 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.status 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-16T23:27:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=92 2024-09-16T23:27:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T23:27:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=38 2024-09-16T23:27:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 92 = 38 2024-09-16T23:27:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T23:27:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=38 2024-09-16T23:27:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 92 = 38 2024-09-16T23:27:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T23:27:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 92 = '' 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.status 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-16T23:27:06.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=92 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 92 = 92 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_4': 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9718118e2 2024-09-16T23:27:07.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9718118e2 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:10 2024 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:10 2024 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:10 2024 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8d46cddb-0d98-4f2b-8e9f-54007c183e08 2024-09-16T23:27:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:07.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.export 2024-09-16T23:27:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.export 2024-09-16T23:27:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.export 2024-09-16T23:27:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.export 2024-09-16T23:27:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.export 2024-09-16T23:27:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.export 2024-09-16T23:27:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_4.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_4.export 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_5 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:08.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:08.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:09.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-16T23:27:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-16T23:27:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-16T23:27:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:09.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:09.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-16T23:27:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:09.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_5 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_5 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_5.status 2024-09-16T23:27:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.status 2024-09-16T23:27:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_5.status 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-16T23:27:10.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 94 2024-09-16T23:27:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.status 2024-09-16T23:27:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.status 2024-09-16T23:27:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-16T23:27:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=94 2024-09-16T23:27:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T23:27:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=39 2024-09-16T23:27:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 94 = 39 2024-09-16T23:27:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:10.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T23:27:11.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:11.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=39 2024-09-16T23:27:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 94 = 39 2024-09-16T23:27:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T23:27:12.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 94 = '' 2024-09-16T23:27:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:13.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.status 2024-09-16T23:27:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-16T23:27:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=94 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 94 = 94 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-16T23:27:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_5': 2024-09-16T23:27:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c942154504 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c942154504 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:21 2024 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:21 2024 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:21 2024 2024-09-16T23:27:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: f84d3a0f-2e8c-472f-b6aa-47f1ebb35940 2024-09-16T23:27:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:14.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.export 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.export 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.export 2024-09-16T23:27:14.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.export 2024-09-16T23:27:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.export 2024-09-16T23:27:15.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.export 2024-09-16T23:27:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_5.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_5.export 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_6 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-16T23:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:15.652 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-16T23:27:16.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:16.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:16.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:16.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-16T23:27:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-16T23:27:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:16.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:16.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-16T23:27:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_6 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_6 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_6.status 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.status 2024-09-16T23:27:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-16T23:27:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_6.status 2024-09-16T23:27:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-16T23:27:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:16.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-16T23:27:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 96 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.status 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.status 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-16T23:27:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=96 2024-09-16T23:27:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:17.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T23:27:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=40 2024-09-16T23:27:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 96 = 40 2024-09-16T23:27:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T23:27:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=40 2024-09-16T23:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 96 = 40 2024-09-16T23:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T23:27:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:19.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:19.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 96 = '' 2024-09-16T23:27:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.status 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-16T23:27:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:20.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=96 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 96 = 96 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:20.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:21.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-16T23:27:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_6': 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c95015685 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c95015685 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:27 2024 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:27 2024 2024-09-16T23:27:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:27 2024 2024-09-16T23:27:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c94c7221-b8a2-4e38-a50c-e34a1f45df1c 2024-09-16T23:27:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.export 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.export 2024-09-16T23:27:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.export 2024-09-16T23:27:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.export 2024-09-16T23:27:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.export 2024-09-16T23:27:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.export 2024-09-16T23:27:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_6.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_6.export 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_7 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-16T23:27:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:22.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:23.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:23.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:23.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:23.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:23.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:23.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:23.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-16T23:27:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2024-09-16T23:27:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:23.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2024-09-16T23:27:23.907 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_7 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_7 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:23.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_7.status 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.status 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_7.status 2024-09-16T23:27:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:23.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-16T23:27:24.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 98 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.status 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.status 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-16T23:27:24.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=98 2024-09-16T23:27:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:24.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T23:27:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=41 2024-09-16T23:27:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 98 = 41 2024-09-16T23:27:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:24.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T23:27:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=41 2024-09-16T23:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 98 = 41 2024-09-16T23:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:25.570 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5160609974463819) > weight (0.25) 2024-09-16T23:27:25.571 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7933216131440588) > weight (0.25) 2024-09-16T23:27:25.571 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.27194510344937706) > weight (0.25) 2024-09-16T23:27:25.571 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5273024668647327) > weight (0.25) 2024-09-16T23:27:25.571 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T23:27:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 98 = '' 2024-09-16T23:27:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.status 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-16T23:27:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:27.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=98 2024-09-16T23:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 98 = 98 2024-09-16T23:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-16T23:27:27.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-16T23:27:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_7': 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c98b43ef90 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c98b43ef90 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:33 2024 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:33 2024 2024-09-16T23:27:29.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:33 2024 2024-09-16T23:27:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8e4da14b-dfc0-4158-b26e-7af55852456e 2024-09-16T23:27:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-16T23:27:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:29.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.export 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.export 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.export 2024-09-16T23:27:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.export 2024-09-16T23:27:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:29.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.export 2024-09-16T23:27:29.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.export 2024-09-16T23:27:29.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_7.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_7.export 2024-09-16T23:27:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_8 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:29.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:29.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:30.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:30.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-16T23:27:30.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2024-09-16T23:27:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:30.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2024-09-16T23:27:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_8 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_8 2024-09-16T23:27:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_8.status 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.status 2024-09-16T23:27:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-16T23:27:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_8.status 2024-09-16T23:27:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-16T23:27:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-16T23:27:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 100 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.status 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.status 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-16T23:27:31.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:31.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=100 2024-09-16T23:27:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T23:27:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:32.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=42 2024-09-16T23:27:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 100 = 42 2024-09-16T23:27:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T23:27:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=42 2024-09-16T23:27:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 100 = 42 2024-09-16T23:27:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T23:27:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:33.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 100 = '' 2024-09-16T23:27:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:34.678 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:27:34.676+0000 7f4bc8b61640 -1 reset not still connected to 0x56270af3d6c0 2024-09-16T23:27:34.679 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:27:34.676+0000 7f4bc8b61640 -1 reset not still connected to 0x56270b479380 2024-09-16T23:27:34.679 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:27:34.676+0000 7f4bc8b61640 -1 reset not still connected to 0x56270dbfa410 2024-09-16T23:27:34.679 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:27:34.676+0000 7f4bc8b61640 -1 reset not still connected to 0x56270dbfa680 2024-09-16T23:27:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.status 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-16T23:27:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=100 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 100 = 100 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:34.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:35.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_8': 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c917cec141 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c917cec141 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:44 2024 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:44 2024 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:44 2024 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: bd37f003-9696-4966-8734-3bc10187a7c9 2024-09-16T23:27:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.export 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.export 2024-09-16T23:27:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.export 2024-09-16T23:27:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.export 2024-09-16T23:27:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.export 2024-09-16T23:27:36.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.export 2024-09-16T23:27:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_8.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_8.export 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_9 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-16T23:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-16T23:27:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:37.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:37.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:37.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-16T23:27:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-16T23:27:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:37.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-16T23:27:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:37.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-16T23:27:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_9 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:37.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_9 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_9.status 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.status 2024-09-16T23:27:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-16T23:27:37.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_9.status 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-16T23:27:38.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 102 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.status 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.status 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-16T23:27:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=102 2024-09-16T23:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T23:27:38.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=43 2024-09-16T23:27:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 102 = 43 2024-09-16T23:27:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T23:27:39.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=43 2024-09-16T23:27:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 102 = 43 2024-09-16T23:27:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T23:27:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 102 = '' 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.status 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-16T23:27:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=102 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 102 = 102 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:41.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-16T23:27:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_9': 2024-09-16T23:27:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9a4711c80 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9a4711c80 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:18:58 2024 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:18:58 2024 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:18:58 2024 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 89e925c9-d5c8-42fb-9de9-72a27fc3d8b5 2024-09-16T23:27:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.export 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.export 2024-09-16T23:27:43.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.export 2024-09-16T23:27:43.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.export 2024-09-16T23:27:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.export 2024-09-16T23:27:43.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.export 2024-09-16T23:27:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_9.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_9.export 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_10 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:44.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:44.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-16T23:27:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-16T23:27:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:44.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:44.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-16T23:27:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:44.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_10 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_10 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-16T23:27:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_10.status 2024-09-16T23:27:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.status 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_10.status 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-16T23:27:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 104 2024-09-16T23:27:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.status 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.status 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-16T23:27:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:45.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=104 2024-09-16T23:27:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:45.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T23:27:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:46.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=44 2024-09-16T23:27:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 104 = 44 2024-09-16T23:27:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T23:27:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=44 2024-09-16T23:27:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 104 = 44 2024-09-16T23:27:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T23:27:47.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:47.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 104 = '' 2024-09-16T23:27:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:48.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.status 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-16T23:27:48.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:48.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=104 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 104 = 104 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-16T23:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_10': 2024-09-16T23:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c93e84a079 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c93e84a079 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:10 2024 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:10 2024 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:10 2024 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ec61c977-0c0f-41e7-8a23-423f6faad7dc 2024-09-16T23:27:50.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-16T23:27:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:50.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.export 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.export 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.export 2024-09-16T23:27:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.export 2024-09-16T23:27:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.export 2024-09-16T23:27:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.export 2024-09-16T23:27:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_10.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_10.export 2024-09-16T23:27:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_11 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:51.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-16T23:27:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-16T23:27:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-16T23:27:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-16T23:27:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_11 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_11 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_11.status 2024-09-16T23:27:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.status 2024-09-16T23:27:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_11.status 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:52.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-16T23:27:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 106 2024-09-16T23:27:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.status 2024-09-16T23:27:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.status 2024-09-16T23:27:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-16T23:27:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=106 2024-09-16T23:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:27:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T23:27:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=45 2024-09-16T23:27:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 106 = 45 2024-09-16T23:27:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T23:27:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 106 = '' 2024-09-16T23:27:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:54.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T23:27:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:27:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:27:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:27:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:27:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:27:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 106 = '' 2024-09-16T23:27:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:27:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:27:55.388 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9191382569364052) > weight (0.25) 2024-09-16T23:27:55.388 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.540904301622764) > weight (0.25) 2024-09-16T23:27:55.388 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T23:27:55.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T23:27:55.389 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T23:27:55.389 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:27:55.392+0000 7fb648240640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 48177) UID: 1000 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.status 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-16T23:27:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:27:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=106 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 106 = 106 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:57.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:27:57.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:58.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_11': 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9aeaf0a0 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9aeaf0a0 2024-09-16T23:27:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:21 2024 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:21 2024 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:21 2024 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 268e0ae9-2b9e-4a82-8b83-58d27f2a3cd5 2024-09-16T23:27:58.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.export 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.export 2024-09-16T23:27:58.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.export 2024-09-16T23:27:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.export 2024-09-16T23:27:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.export 2024-09-16T23:27:58.390 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T23:27:58.390 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:27:58.390 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T23:27:58.390 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:27:58.391 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:27:58.393 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:27:58.393 INFO:tasks.rbd_mirror_thrash:waiting for 111 secs before thrashing 2024-09-16T23:27:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:27:58.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.export 2024-09-16T23:27:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_11.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_11.export 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_12 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:27:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:27:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:27:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:27:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:59.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:27:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:27:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:27:59.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:27:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:27:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-16T23:27:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2024-09-16T23:27:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:27:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:27:59.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:27:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:27:59.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:27:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:27:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2024-09-16T23:27:59.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:27:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:27:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:27:59.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:27:59.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_12 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_12 2024-09-16T23:27:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_12.status 2024-09-16T23:27:59.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.status 2024-09-16T23:27:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-16T23:27:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_12.status 2024-09-16T23:27:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:27:59.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:27:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-16T23:27:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:27:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:27:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:27:59.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-16T23:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 108 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.status 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.status 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-16T23:28:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=108 2024-09-16T23:28:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:00.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T23:28:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:01.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=46 2024-09-16T23:28:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 108 = 46 2024-09-16T23:28:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:01.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T23:28:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 108 = '' 2024-09-16T23:28:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T23:28:02.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:02.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:02.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 108 = '' 2024-09-16T23:28:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:02.515 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.517+0000 7fbcef505640 -1 reset not still connected to 0x56517d193930 2024-09-16T23:28:02.516 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.518+0000 7fbcef505640 -1 reset not still connected to 0x56517d1d09c0 2024-09-16T23:28:02.537 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.536+0000 7fbcef505640 -1 reset not still connected to 0x56517d6b1110 2024-09-16T23:28:02.537 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.536+0000 7fbcef505640 -1 reset not still connected to 0x56517e64b110 2024-09-16T23:28:02.537 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.536+0000 7fbcef505640 -1 reset not still connected to 0x5651808dcb60 2024-09-16T23:28:02.537 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.536+0000 7fbcef505640 -1 reset not still connected to 0x565180d9f2b0 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.536+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a6680 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.537+0000 7fbcef505640 -1 reset not still connected to 0x5651928a6000 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.537+0000 7fbcef505640 -1 reset not still connected to 0x5651928a64e0 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.537+0000 7fbcef505640 -1 reset not still connected to 0x56519be44270 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.537+0000 7fbcef505640 -1 reset not still connected to 0x56519c3a3110 2024-09-16T23:28:02.538 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:28:02.537+0000 7fbcef505640 -1 reset not still connected to 0x56519c3a3c70 2024-09-16T23:28:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.status 2024-09-16T23:28:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-16T23:28:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=108 2024-09-16T23:28:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 108 = 108 2024-09-16T23:28:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:04.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:04.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:05.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-16T23:28:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_12': 2024-09-16T23:28:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9a45229e 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9a45229e 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:34 2024 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:34 2024 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:34 2024 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b9122161-1c25-422e-ae7b-083efda568a9 2024-09-16T23:28:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.export 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.export 2024-09-16T23:28:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.export 2024-09-16T23:28:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.export 2024-09-16T23:28:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.export 2024-09-16T23:28:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.export 2024-09-16T23:28:05.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_12.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_12.export 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_13 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-16T23:28:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-16T23:28:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:05.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-16T23:28:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-16T23:28:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2024-09-16T23:28:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:06.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:06.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:06.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2024-09-16T23:28:06.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_13 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_13 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_13.status 2024-09-16T23:28:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.status 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_13.status 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-16T23:28:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 110 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.status 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.status 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-16T23:28:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=110 2024-09-16T23:28:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T23:28:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=47 2024-09-16T23:28:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 110 = 47 2024-09-16T23:28:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T23:28:08.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 110 = '' 2024-09-16T23:28:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:09.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T23:28:09.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:09.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 110 = '' 2024-09-16T23:28:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.status 2024-09-16T23:28:10.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-16T23:28:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=110 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 110 = 110 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:11.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:11.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-16T23:28:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_13': 2024-09-16T23:28:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c92cccd59a 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c92cccd59a 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:40 2024 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:40 2024 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:40 2024 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b92b2830-e993-4cb1-800e-5126fdb3887f 2024-09-16T23:28:12.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-16T23:28:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.export 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.export 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.export 2024-09-16T23:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.export 2024-09-16T23:28:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.export 2024-09-16T23:28:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:12.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.export 2024-09-16T23:28:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_13.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_13.export 2024-09-16T23:28:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_14 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-16T23:28:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-16T23:28:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-16T23:28:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-16T23:28:13.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:13.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-16T23:28:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_14 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_14 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_14.status 2024-09-16T23:28:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.status 2024-09-16T23:28:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-16T23:28:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_14.status 2024-09-16T23:28:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-16T23:28:14.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 112 2024-09-16T23:28:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.status 2024-09-16T23:28:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.status 2024-09-16T23:28:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-16T23:28:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=112 2024-09-16T23:28:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T23:28:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=48 2024-09-16T23:28:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 112 = 48 2024-09-16T23:28:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T23:28:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 112 = '' 2024-09-16T23:28:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T23:28:16.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:16.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 112 = '' 2024-09-16T23:28:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:18.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.status 2024-09-16T23:28:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-16T23:28:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=112 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 112 = 112 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:19.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_14': 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c990a8e986 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c990a8e986 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:19.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:47 2024 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:47 2024 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:47 2024 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ca2b9bfb-9340-437f-9dcd-274b942d579d 2024-09-16T23:28:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.export 2024-09-16T23:28:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.export 2024-09-16T23:28:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.export 2024-09-16T23:28:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.export 2024-09-16T23:28:19.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:19.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.export 2024-09-16T23:28:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.export 2024-09-16T23:28:19.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_14.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_14.export 2024-09-16T23:28:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_15 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:20.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:20.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-16T23:28:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:21.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:21.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-16T23:28:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-16T23:28:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:21.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:21.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:21.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-16T23:28:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:22.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_15 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:22.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_15 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_15.status 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.status 2024-09-16T23:28:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-16T23:28:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_15.status 2024-09-16T23:28:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-16T23:28:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 114 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.status 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.status 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-16T23:28:22.788 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=114 2024-09-16T23:28:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:22.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:23.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T23:28:23.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=49 2024-09-16T23:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 114 = 49 2024-09-16T23:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:23.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T23:28:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=49 2024-09-16T23:28:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 114 = 49 2024-09-16T23:28:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:23.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T23:28:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 114 = '' 2024-09-16T23:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.status 2024-09-16T23:28:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-16T23:28:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=114 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 114 = 114 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_15': 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c94227c895 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c94227c895 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:19:56 2024 2024-09-16T23:28:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:19:56 2024 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:19:56 2024 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: d611b2eb-d352-4abb-977d-a94f77c573c9 2024-09-16T23:28:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.export 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.export 2024-09-16T23:28:27.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.export 2024-09-16T23:28:27.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.export 2024-09-16T23:28:27.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.export 2024-09-16T23:28:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.export 2024-09-16T23:28:27.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_15.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_15.export 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_16 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-16T23:28:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:27.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:27.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-16T23:28:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-16T23:28:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-16T23:28:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-16T23:28:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-16T23:28:28.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-16T23:28:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_16 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_16 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-16T23:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_16.status 2024-09-16T23:28:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.status 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_16.status 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-16T23:28:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 116 2024-09-16T23:28:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.status 2024-09-16T23:28:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.status 2024-09-16T23:28:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-16T23:28:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=116 2024-09-16T23:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:29.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T23:28:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=50 2024-09-16T23:28:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 116 = 50 2024-09-16T23:28:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:30.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T23:28:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 116 = '' 2024-09-16T23:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T23:28:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 116 = '' 2024-09-16T23:28:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.status 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-16T23:28:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=116 2024-09-16T23:28:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 116 = 116 2024-09-16T23:28:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:34.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-16T23:28:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_16': 2024-09-16T23:28:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9e1f7f7f0 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9e1f7f7f0 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:04 2024 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:04 2024 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:04 2024 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 161cd388-c85e-48c4-8c8e-84aaca587105 2024-09-16T23:28:34.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:34.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.export 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.export 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.export 2024-09-16T23:28:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.export 2024-09-16T23:28:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:34.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.export 2024-09-16T23:28:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:34.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.export 2024-09-16T23:28:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_16.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_16.export 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_17 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:34.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:34.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-16T23:28:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-16T23:28:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:35.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:35.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:35.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-16T23:28:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2024-09-16T23:28:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:35.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:35.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:35.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2024-09-16T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:36.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:36.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_17 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_17 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:36.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-16T23:28:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_17.status 2024-09-16T23:28:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.status 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_17.status 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-16T23:28:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 118 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.status 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.status 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-16T23:28:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=118 2024-09-16T23:28:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T23:28:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:37.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=51 2024-09-16T23:28:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 118 = 51 2024-09-16T23:28:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T23:28:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 118 = '' 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T23:28:38.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 118 = '' 2024-09-16T23:28:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.status 2024-09-16T23:28:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-16T23:28:40.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=118 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 118 = 118 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:40.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-16T23:28:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_17': 2024-09-16T23:28:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9c2c813a 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9c2c813a 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:15 2024 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:15 2024 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:15 2024 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 77e8cc56-9128-4126-9b3a-02b903130874 2024-09-16T23:28:41.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:41.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.export 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.export 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.export 2024-09-16T23:28:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.export 2024-09-16T23:28:41.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.export 2024-09-16T23:28:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:41.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.export 2024-09-16T23:28:41.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_17.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_17.export 2024-09-16T23:28:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_18 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:42.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:43.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-16T23:28:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:43.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:43.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-16T23:28:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:43.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:43.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:43.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-16T23:28:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-16T23:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-16T23:28:44.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:44.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_18 2024-09-16T23:28:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_18 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-16T23:28:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_18.status 2024-09-16T23:28:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.status 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_18.status 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-16T23:28:44.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 120 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.status 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.status 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-16T23:28:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=120 2024-09-16T23:28:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:44.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T23:28:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=52 2024-09-16T23:28:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 120 = 52 2024-09-16T23:28:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T23:28:45.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:45.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 120 = '' 2024-09-16T23:28:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:45.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T23:28:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 120 = '' 2024-09-16T23:28:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:46.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:47.604 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:28:47.606+0000 7f8c0afcb640 -1 reset not still connected to 0x558f11fa9ad0 2024-09-16T23:28:47.607 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:28:47.609+0000 7f8c0afcb640 -1 reset not still connected to 0x558f16f3b2b0 2024-09-16T23:28:48.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.status 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-16T23:28:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:48.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=120 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 120 = 120 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-16T23:28:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_18': 2024-09-16T23:28:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9360f36c3 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9360f36c3 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:30 2024 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:30 2024 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:30 2024 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:49.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 58af7715-4fc6-402e-aa27-d73e969a9d7a 2024-09-16T23:28:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-16T23:28:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.export 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.export 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.export 2024-09-16T23:28:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.export 2024-09-16T23:28:49.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.export 2024-09-16T23:28:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.export 2024-09-16T23:28:49.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_18.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_18.export 2024-09-16T23:28:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_19 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-16T23:28:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:50.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:50.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:50.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-16T23:28:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2024-09-16T23:28:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2024-09-16T23:28:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_19 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_19 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_19.status 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.status 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-16T23:28:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_19.status 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-16T23:28:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 122 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.status 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.status 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-16T23:28:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=122 2024-09-16T23:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T23:28:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=53 2024-09-16T23:28:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 122 = 53 2024-09-16T23:28:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T23:28:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:52.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 122 = '' 2024-09-16T23:28:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:28:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T23:28:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 122 = '' 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:28:54.627 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.627+0000 7f4bc8b61640 -1 reset not still connected to 0x562708d08f70 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x562709612750 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x562709613d40 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270984d040 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270aa5dad0 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270aa5dee0 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270af3c4e0 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270af3ca90 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270af3d6c0 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270b478340 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270b479380 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270dbfa410 2024-09-16T23:28:54.631 INFO:tasks.ceph.cluster1.osd.2.smithi096.stderr:2024-09-16T23:28:54.630+0000 7f4bc8b61640 -1 reset not still connected to 0x56270dbfa680 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.status 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-16T23:28:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=122 2024-09-16T23:28:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 122 = 122 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:28:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_19': 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c94b5e9488 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c94b5e9488 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:28:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:40 2024 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:40 2024 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:40 2024 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 76b07c29-6ac0-440f-9c18-26ac80777711 2024-09-16T23:28:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:28:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-16T23:28:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.export 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.export 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.export 2024-09-16T23:28:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.export 2024-09-16T23:28:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.export 2024-09-16T23:28:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:28:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.export 2024-09-16T23:28:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_19.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_19.export 2024-09-16T23:28:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_20 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:28:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:28:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-16T23:28:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:56.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:28:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-16T23:28:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:28:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:28:57.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:28:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:28:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:28:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:28:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:28:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:28:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:28:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:28:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:28:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-16T23:28:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-16T23:28:57.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:28:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:28:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:28:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:28:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:28:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:28:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-16T23:28:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:28:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:28:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:28:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:28:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:28:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:28:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:28:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:28:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_20 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_20 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_20.status 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.status 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_20.status 2024-09-16T23:28:58.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-16T23:28:58.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 124 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.status 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.status 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-16T23:28:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:28:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=124 2024-09-16T23:28:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T23:28:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=54 2024-09-16T23:28:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 124 = 54 2024-09-16T23:28:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:28:59.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T23:28:59.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:28:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:28:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:28:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:28:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:28:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:28:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:28:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 124 = '' 2024-09-16T23:28:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:28:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T23:29:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 124 = '' 2024-09-16T23:29:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:00.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.status 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-16T23:29:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=124 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 124 = 124 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:02.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:29:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-16T23:29:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_20': 2024-09-16T23:29:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c98f4bc210 2024-09-16T23:29:03.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c98f4bc210 2024-09-16T23:29:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:03.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:20:46 2024 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:20:46 2024 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:20:46 2024 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3e27fd33-23b8-4a4b-a8ee-9baf2df8b8f0 2024-09-16T23:29:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:03.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-16T23:29:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:29:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.export 2024-09-16T23:29:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.export 2024-09-16T23:29:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.export 2024-09-16T23:29:03.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.export 2024-09-16T23:29:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.export 2024-09-16T23:29:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:03.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.export 2024-09-16T23:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_20.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_20.export 2024-09-16T23:29:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_21 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:04.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-16T23:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-16T23:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-16T23:29:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_21 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_21 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_21.status 2024-09-16T23:29:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.status 2024-09-16T23:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-16T23:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_21.status 2024-09-16T23:29:06.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:06.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-16T23:29:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 126 2024-09-16T23:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.status 2024-09-16T23:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.status 2024-09-16T23:29:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-16T23:29:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=126 2024-09-16T23:29:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T23:29:07.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=55 2024-09-16T23:29:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 126 = 55 2024-09-16T23:29:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:07.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T23:29:07.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 126 = '' 2024-09-16T23:29:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:08.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7150589752009677) > weight (0.25) 2024-09-16T23:29:08.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8597276546296649) > weight (0.25) 2024-09-16T23:29:08.572 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6148641176948718) > weight (0.25) 2024-09-16T23:29:08.572 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-16T23:29:08.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-16T23:29:08.573 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-16T23:29:08.574 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:29:08.576+0000 7f98ed91f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 50056) UID: 1000 2024-09-16T23:29:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:08.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:08.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T23:29:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 126 = '' 2024-09-16T23:29:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.status 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-16T23:29:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=126 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 126 = 126 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_21': 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9ba3e4f34 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9ba3e4f34 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:01 2024 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:01 2024 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:01 2024 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: eecd37c8-2507-4ea3-8c85-72ea35b897c6 2024-09-16T23:29:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.export 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.export 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.export 2024-09-16T23:29:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.export 2024-09-16T23:29:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.export 2024-09-16T23:29:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.export 2024-09-16T23:29:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_21.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_21.export 2024-09-16T23:29:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_22 2024-09-16T23:29:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-16T23:29:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-16T23:29:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-16T23:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-16T23:29:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:14.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:14.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_22 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_22 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_22.status 2024-09-16T23:29:14.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.status 2024-09-16T23:29:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-16T23:29:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_22.status 2024-09-16T23:29:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-16T23:29:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-16T23:29:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 128 2024-09-16T23:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.status 2024-09-16T23:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.status 2024-09-16T23:29:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-16T23:29:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=128 2024-09-16T23:29:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:14.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T23:29:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=56 2024-09-16T23:29:15.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 128 = 56 2024-09-16T23:29:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:15.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:15.574 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-16T23:29:15.574 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:29:15.575 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-16T23:29:15.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-16T23:29:15.575 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-16T23:29:15.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-16T23:29:15.578 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T23:29:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 128 = '' 2024-09-16T23:29:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T23:29:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:16.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 128 = '' 2024-09-16T23:29:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:16.579 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.status 2024-09-16T23:29:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-16T23:29:18.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=128 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 128 = 128 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-16T23:29:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_22': 2024-09-16T23:29:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c928b80e10 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c928b80e10 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:15 2024 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:15 2024 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:15 2024 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: fb737a24-defd-44de-8405-d2f600fcfa9d 2024-09-16T23:29:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.export 2024-09-16T23:29:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.export 2024-09-16T23:29:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.export 2024-09-16T23:29:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.export 2024-09-16T23:29:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:19.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.export 2024-09-16T23:29:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.export 2024-09-16T23:29:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_22.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_22.export 2024-09-16T23:29:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_23 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:19.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:20.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:20.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:20.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-16T23:29:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2024-09-16T23:29:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:21.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:21.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2024-09-16T23:29:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_23 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_23 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-16T23:29:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_23.status 2024-09-16T23:29:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.status 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_23.status 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-16T23:29:21.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 130 2024-09-16T23:29:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.status 2024-09-16T23:29:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:21.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.status 2024-09-16T23:29:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-16T23:29:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=130 2024-09-16T23:29:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T23:29:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=57 2024-09-16T23:29:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 130 = 57 2024-09-16T23:29:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T23:29:22.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517ca6cdd0 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517cf789c0 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517cf791e0 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517d192c30 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517d193930 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517d1d09c0 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517d6b1110 2024-09-16T23:29:22.692 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.693+0000 7fbcef505640 -1 reset not still connected to 0x56517e64b110 2024-09-16T23:29:22.697 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x5651808dcb60 2024-09-16T23:29:22.697 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x565180d9f2b0 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a6680 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x5651928a6000 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x5651928a64e0 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x56519be44270 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x56519c3a3110 2024-09-16T23:29:22.698 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:29:22.699+0000 7fbcef505640 -1 reset not still connected to 0x56519c3a3c70 2024-09-16T23:29:22.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 130 = '' 2024-09-16T23:29:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T23:29:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 130 = '' 2024-09-16T23:29:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:23.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.status 2024-09-16T23:29:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-16T23:29:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=130 2024-09-16T23:29:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 130 = 130 2024-09-16T23:29:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-16T23:29:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_23': 2024-09-16T23:29:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9ff42623c 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9ff42623c 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:28 2024 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:28 2024 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:28 2024 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: e1d61309-9267-41b0-b201-a6011fded361 2024-09-16T23:29:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.export 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.export 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.export 2024-09-16T23:29:26.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.export 2024-09-16T23:29:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.export 2024-09-16T23:29:26.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.export 2024-09-16T23:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_23.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_23.export 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_24 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-16T23:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-16T23:29:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:27.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-16T23:29:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:27.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-16T23:29:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2024-09-16T23:29:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:28.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:28.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2024-09-16T23:29:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_24 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_24 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_24.status 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.status 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_24.status 2024-09-16T23:29:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:28.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-16T23:29:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 132 2024-09-16T23:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.status 2024-09-16T23:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.status 2024-09-16T23:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-16T23:29:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=132 2024-09-16T23:29:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T23:29:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:29.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=58 2024-09-16T23:29:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 132 = 58 2024-09-16T23:29:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T23:29:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=58 2024-09-16T23:29:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 132 = 58 2024-09-16T23:29:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:30.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T23:29:30.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 132 = '' 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.status 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-16T23:29:32.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=132 2024-09-16T23:29:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 132 = 132 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_24': 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c959e23227 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c959e23227 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:33.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:35 2024 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:35 2024 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:35 2024 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 449d20a2-3295-43fc-99a8-bae9ff72cca1 2024-09-16T23:29:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-16T23:29:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.export 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.export 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.export 2024-09-16T23:29:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.export 2024-09-16T23:29:33.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:33.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.export 2024-09-16T23:29:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.export 2024-09-16T23:29:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_24.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_24.export 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_25 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:33.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-16T23:29:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:34.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-16T23:29:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2024-09-16T23:29:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:35.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2024-09-16T23:29:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_25 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_25 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-16T23:29:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_25.status 2024-09-16T23:29:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.status 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_25.status 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-16T23:29:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 134 2024-09-16T23:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.status 2024-09-16T23:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.status 2024-09-16T23:29:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-16T23:29:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=134 2024-09-16T23:29:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T23:29:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=59 2024-09-16T23:29:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 134 = 59 2024-09-16T23:29:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T23:29:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=59 2024-09-16T23:29:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 134 = 59 2024-09-16T23:29:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T23:29:37.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 134 = '' 2024-09-16T23:29:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.status 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-16T23:29:39.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=134 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 134 = 134 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:39.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:39.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-16T23:29:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_25': 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9f0c15b3c 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9f0c15b3c 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:45 2024 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:45 2024 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:45 2024 2024-09-16T23:29:40.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: aa545518-746e-469d-a87e-2348440b9692 2024-09-16T23:29:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.export 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.export 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.export 2024-09-16T23:29:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.export 2024-09-16T23:29:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.export 2024-09-16T23:29:40.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:40.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.export 2024-09-16T23:29:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_25.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_25.export 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_26 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:40.948 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-16T23:29:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:41.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-16T23:29:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-16T23:29:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:42.058 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-16T23:29:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_26 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_26 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_26.status 2024-09-16T23:29:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.status 2024-09-16T23:29:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-16T23:29:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_26.status 2024-09-16T23:29:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-16T23:29:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 136 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.status 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.status 2024-09-16T23:29:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-16T23:29:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=136 2024-09-16T23:29:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T23:29:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=60 2024-09-16T23:29:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 136 = 60 2024-09-16T23:29:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T23:29:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=60 2024-09-16T23:29:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 136 = 60 2024-09-16T23:29:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T23:29:44.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:44.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 136 = '' 2024-09-16T23:29:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:44.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:46.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.status 2024-09-16T23:29:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-16T23:29:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:46.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=136 2024-09-16T23:29:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 136 = 136 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-16T23:29:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_26': 2024-09-16T23:29:47.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9c7b5937d 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9c7b5937d 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:21:54 2024 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:21:54 2024 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:21:54 2024 2024-09-16T23:29:47.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3a054c79-09d1-4ef0-b9c5-083578f40968 2024-09-16T23:29:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-16T23:29:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:29:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.export 2024-09-16T23:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.export 2024-09-16T23:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.export 2024-09-16T23:29:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.export 2024-09-16T23:29:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.export 2024-09-16T23:29:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.export 2024-09-16T23:29:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_26.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_26.export 2024-09-16T23:29:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_27 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:47.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-16T23:29:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:48.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:48.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-16T23:29:49.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-16T23:29:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-16T23:29:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-16T23:29:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_27 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:49.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_27 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_27.status 2024-09-16T23:29:49.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.status 2024-09-16T23:29:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-16T23:29:49.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_27.status 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:49.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-16T23:29:49.394 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.529329998772432) > weight (0.25) 2024-09-16T23:29:49.394 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3216235191236454) > weight (0.25) 2024-09-16T23:29:49.394 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8895040566951414) > weight (0.25) 2024-09-16T23:29:49.395 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7155236606948776) > weight (0.25) 2024-09-16T23:29:49.395 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-16T23:29:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 138 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.status 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.status 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-16T23:29:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:50.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=138 2024-09-16T23:29:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:50.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T23:29:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:50.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=61 2024-09-16T23:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 138 = 61 2024-09-16T23:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:50.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:50.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T23:29:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:50.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=61 2024-09-16T23:29:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 138 = 61 2024-09-16T23:29:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:51.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:51.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T23:29:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:51.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 138 = '' 2024-09-16T23:29:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:29:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:53.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.status 2024-09-16T23:29:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-16T23:29:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=138 2024-09-16T23:29:53.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 138 = 138 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:29:53.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-16T23:29:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_27': 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9105c8601 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9105c8601 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:07 2024 2024-09-16T23:29:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:07 2024 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:07 2024 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 0de0d1ea-4ea9-47eb-a8fc-cc4b4c55433f 2024-09-16T23:29:54.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.export 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.export 2024-09-16T23:29:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.export 2024-09-16T23:29:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.export 2024-09-16T23:29:54.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.export 2024-09-16T23:29:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:29:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.export 2024-09-16T23:29:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_27.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_27.export 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_28 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:29:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:29:55.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:29:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-16T23:29:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:29:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:29:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:29:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:29:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:29:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:29:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:29:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-16T23:29:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-16T23:29:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:29:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:29:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:29:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:29:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:29:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:29:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-16T23:29:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:29:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:29:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:29:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:29:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:29:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_28 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:29:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_28 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_28.status 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.status 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_28.status 2024-09-16T23:29:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-16T23:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-16T23:29:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 140 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.status 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.status 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-16T23:29:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:29:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=140 2024-09-16T23:29:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:29:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T23:29:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=62 2024-09-16T23:29:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 140 = 62 2024-09-16T23:29:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T23:29:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 140 = '' 2024-09-16T23:29:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T23:29:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:29:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:29:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:29:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:29:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:29:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:29:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:29:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 140 = '' 2024-09-16T23:29:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:29:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:30:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:30:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:30:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.status 2024-09-16T23:30:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-16T23:30:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:00.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:00.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=140 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 140 = 140 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_28': 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c94065b605 2024-09-16T23:30:01.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c94065b605 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:16 2024 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:16 2024 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:16 2024 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 77ea12b7-17e8-4b4a-ad76-3f55bd6a6279 2024-09-16T23:30:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.export 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.export 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.export 2024-09-16T23:30:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.export 2024-09-16T23:30:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.export 2024-09-16T23:30:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:01.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.export 2024-09-16T23:30:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_28.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_28.export 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_29 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:01.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-16T23:30:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-16T23:30:03.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:03.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-16T23:30:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-16T23:30:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:03.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-16T23:30:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:03.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_29 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_29 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_29.status 2024-09-16T23:30:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.status 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_29.status 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:03.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-16T23:30:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 142 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.status 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.status 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-16T23:30:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=142 2024-09-16T23:30:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T23:30:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=63 2024-09-16T23:30:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 142 = 63 2024-09-16T23:30:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T23:30:04.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 142 = '' 2024-09-16T23:30:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T23:30:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:05.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 142 = '' 2024-09-16T23:30:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.status 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-16T23:30:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=142 2024-09-16T23:30:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 142 = 142 2024-09-16T23:30:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-16T23:30:07.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:08.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-16T23:30:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_29': 2024-09-16T23:30:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9af517c8d 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9af517c8d 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:29 2024 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:29 2024 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:29 2024 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: d119fabc-3461-4f14-ad5b-937b0502300c 2024-09-16T23:30:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-16T23:30:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:30:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.export 2024-09-16T23:30:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.export 2024-09-16T23:30:08.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.export 2024-09-16T23:30:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.export 2024-09-16T23:30:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.export 2024-09-16T23:30:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.export 2024-09-16T23:30:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_29.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_29.export 2024-09-16T23:30:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_30 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:08.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:09.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-16T23:30:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:10.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-16T23:30:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-16T23:30:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:10.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:10.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:10.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-16T23:30:10.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:10.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_30 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_30 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_30.status 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.status 2024-09-16T23:30:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-16T23:30:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_30.status 2024-09-16T23:30:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.status 2024-09-16T23:30:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-16T23:30:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-16T23:30:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 144 2024-09-16T23:30:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.status 2024-09-16T23:30:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.status 2024-09-16T23:30:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-16T23:30:11.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=144 2024-09-16T23:30:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:11.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.status 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.status 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T23:30:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:11.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 144 = '' 2024-09-16T23:30:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.status 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.status 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-16T23:30:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=144 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 144 = 144 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-16T23:30:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_30': 2024-09-16T23:30:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9714694a2 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9714694a2 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:37 2024 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:37 2024 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:37 2024 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 9a9a2e05-0891-4a2c-bb16-07ba21d8071f 2024-09-16T23:30:12.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:12.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.export 2024-09-16T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.export 2024-09-16T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.export 2024-09-16T23:30:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.export 2024-09-16T23:30:13.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.export 2024-09-16T23:30:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.export 2024-09-16T23:30:13.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_30.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_30.export 2024-09-16T23:30:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_31 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-16T23:30:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-16T23:30:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:15.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-16T23:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-16T23:30:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:15.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-16T23:30:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_31 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_31 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_31.status 2024-09-16T23:30:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.status 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_31.status 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-16T23:30:16.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 146 2024-09-16T23:30:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.status 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.status 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-16T23:30:16.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=146 2024-09-16T23:30:16.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:16.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T23:30:16.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=65 2024-09-16T23:30:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 146 = 65 2024-09-16T23:30:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T23:30:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:16.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 146 = '' 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T23:30:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:17.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 146 = '' 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.status 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-16T23:30:19.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=146 2024-09-16T23:30:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 146 = 146 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_31': 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9b314b61d 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9b314b61d 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:45 2024 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:45 2024 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:45 2024 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c6104db7-e7bf-4eea-a889-773f358930c7 2024-09-16T23:30:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.export 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.export 2024-09-16T23:30:20.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.export 2024-09-16T23:30:20.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.export 2024-09-16T23:30:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.export 2024-09-16T23:30:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.export 2024-09-16T23:30:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_31.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_31.export 2024-09-16T23:30:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_32 2024-09-16T23:30:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-16T23:30:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:22.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:22.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-16T23:30:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:23.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-16T23:30:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:23.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:23.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:23.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:23.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-16T23:30:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-16T23:30:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:23.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:23.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-16T23:30:23.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:23.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_32 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_32 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:23.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-16T23:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_32.status 2024-09-16T23:30:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.status 2024-09-16T23:30:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-16T23:30:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_32.status 2024-09-16T23:30:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:23.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-16T23:30:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-16T23:30:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 148 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.status 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.status 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-16T23:30:24.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=148 2024-09-16T23:30:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T23:30:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=66 2024-09-16T23:30:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 148 = 66 2024-09-16T23:30:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T23:30:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 148 = '' 2024-09-16T23:30:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:25.607 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.606+0000 7f440f149640 -1 reset not still connected to 0x55f266c46c30 2024-09-16T23:30:25.607 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.606+0000 7f440f149640 -1 reset not still connected to 0x55f266c47110 2024-09-16T23:30:25.607 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.607+0000 7f440f149640 -1 reset not still connected to 0x55f269ec25b0 2024-09-16T23:30:25.608 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.609+0000 7f440f149640 -1 reset not still connected to 0x55f26b1029c0 2024-09-16T23:30:25.608 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.609+0000 7f440f149640 -1 reset not still connected to 0x55f26f363380 2024-09-16T23:30:25.620 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.621+0000 7f440f149640 -1 reset not still connected to 0x55f281d6b5f0 2024-09-16T23:30:25.620 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.621+0000 7f440f149640 -1 reset not still connected to 0x55f285467110 2024-09-16T23:30:25.620 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.621+0000 7f440f149640 -1 reset not still connected to 0x55f2887b09c0 2024-09-16T23:30:25.620 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.621+0000 7f440f149640 -1 reset not still connected to 0x55f289c56680 2024-09-16T23:30:25.620 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:30:25.621+0000 7f440f149640 -1 reset not still connected to 0x55f28eb47ad0 2024-09-16T23:30:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.status 2024-09-16T23:30:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-16T23:30:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=148 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 148 = 148 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:25.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:25.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_32': 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c91cedbcc3 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c91cedbcc3 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:22:53 2024 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:22:53 2024 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:22:53 2024 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 11ebbc36-f469-465c-b15a-fe40b29147fb 2024-09-16T23:30:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.export 2024-09-16T23:30:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.export 2024-09-16T23:30:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.export 2024-09-16T23:30:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.export 2024-09-16T23:30:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:26.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.export 2024-09-16T23:30:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.export 2024-09-16T23:30:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_32.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_32.export 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_33 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-16T23:30:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:29.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:29.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:29.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-16T23:30:29.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-16T23:30:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:29.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-16T23:30:29.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_33 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_33 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_33.status 2024-09-16T23:30:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.status 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_33.status 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-16T23:30:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 150 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.status 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.status 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-16T23:30:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=150 2024-09-16T23:30:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:30.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T23:30:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 150 = '' 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:31.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T23:30:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 150 = '' 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:31.579 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5087119569010776) > weight (0.25) 2024-09-16T23:30:31.579 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-16T23:30:31.580 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-16T23:30:31.580 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-16T23:30:31.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:30:31.582+0000 7fea47380640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 38348) UID: 1000 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.status 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-16T23:30:31.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=150 2024-09-16T23:30:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 150 = 150 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:31.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:32.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_33': 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 3 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9e016ff8c 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9e016ff8c 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:03 2024 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:03 2024 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:03 2024 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: e491a9c0-f499-472f-b800-fabf601f9b18 2024-09-16T23:30:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.export 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.export 2024-09-16T23:30:32.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.export 2024-09-16T23:30:32.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.export 2024-09-16T23:30:33.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.export 2024-09-16T23:30:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.export 2024-09-16T23:30:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_33.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_33.export 2024-09-16T23:30:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_34 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-16T23:30:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-16T23:30:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-16T23:30:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:30:35.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:30:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-16T23:30:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-16T23:30:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:3 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:3 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=3 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=3 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:3 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:3 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=3 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=3 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=3 2024-09-16T23:30:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-16T23:30:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:30:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:30:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-16T23:30:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2024-09-16T23:30:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-16T23:30:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 1 2024-09-16T23:30:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:36.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-16T23:30:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-16T23:30:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:36.575 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:36.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-16T23:30:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:30:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:30:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:30:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:30:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:36.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2024-09-16T23:30:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:36.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:36.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_34 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_34 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-16T23:30:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_34.status 2024-09-16T23:30:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.status 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_34.status 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-16T23:30:37.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 152 2024-09-16T23:30:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.status 2024-09-16T23:30:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:37.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.status 2024-09-16T23:30:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-16T23:30:37.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=152 2024-09-16T23:30:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T23:30:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:37.580 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-16T23:30:37.581 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:30:37.581 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-16T23:30:37.581 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-16T23:30:37.582 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-16T23:30:37.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-16T23:30:37.584 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-16T23:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 152 = '' 2024-09-16T23:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T23:30:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 152 = '' 2024-09-16T23:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.status 2024-09-16T23:30:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-16T23:30:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:38.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=152 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 152 = 152 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:38.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_34': 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9594d888e 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9594d888e 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:12 2024 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:12 2024 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:12 2024 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2dc3b71d-4f9f-4123-bb64-1b41606c25b5 2024-09-16T23:30:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.export 2024-09-16T23:30:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.export 2024-09-16T23:30:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.export 2024-09-16T23:30:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.export 2024-09-16T23:30:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.export 2024-09-16T23:30:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.export 2024-09-16T23:30:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_34.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_34.export 2024-09-16T23:30:40.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_35 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:40.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-16T23:30:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:42.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:42.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-16T23:30:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-16T23:30:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-16T23:30:42.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-16T23:30:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:42.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:42.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:30:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:30:42.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:30:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-16T23:30:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_35 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_35 2024-09-16T23:30:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_35.status 2024-09-16T23:30:42.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.status 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_35.status 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:42.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:42.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-16T23:30:43.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 154 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.status 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.status 2024-09-16T23:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-16T23:30:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=154 2024-09-16T23:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T23:30:43.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=69 2024-09-16T23:30:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 154 = 69 2024-09-16T23:30:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:43.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T23:30:43.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 154 = '' 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.status 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-16T23:30:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=154 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 154 = 154 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:45.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_35': 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c96b281cf2 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c96b281cf2 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:18 2024 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:18 2024 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:18 2024 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: a2742cc4-6203-4618-aa8e-430f3fe6d0e9 2024-09-16T23:30:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.export 2024-09-16T23:30:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.export 2024-09-16T23:30:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.export 2024-09-16T23:30:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.export 2024-09-16T23:30:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.export 2024-09-16T23:30:46.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.export 2024-09-16T23:30:46.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_35.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_35.export 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_36 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-16T23:30:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-16T23:30:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:48.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:48.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-16T23:30:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T23:30:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-16T23:30:48.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:48.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-16T23:30:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-16T23:30:48.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-16T23:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:30:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:30:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-16T23:30:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:48.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:48.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_36 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_36 2024-09-16T23:30:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_36.status 2024-09-16T23:30:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.status 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_36.status 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-16T23:30:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-16T23:30:49.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 156 2024-09-16T23:30:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.status 2024-09-16T23:30:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.status 2024-09-16T23:30:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-16T23:30:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=156 2024-09-16T23:30:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:49.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T23:30:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=70 2024-09-16T23:30:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 156 = 70 2024-09-16T23:30:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T23:30:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 156 = '' 2024-09-16T23:30:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.status 2024-09-16T23:30:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-16T23:30:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=156 2024-09-16T23:30:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 156 = 156 2024-09-16T23:30:50.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-16T23:30:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_36': 2024-09-16T23:30:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9adbe2716 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9adbe2716 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:29 2024 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:29 2024 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:29 2024 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 03dea910-3ab0-46f2-a90e-2ddda005fc50 2024-09-16T23:30:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-16T23:30:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.export 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.export 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.export 2024-09-16T23:30:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.export 2024-09-16T23:30:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.export 2024-09-16T23:30:52.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.export 2024-09-16T23:30:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_36.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_36.export 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_37 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-16T23:30:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:30:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:30:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:30:54.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:54.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:30:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:30:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:30:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-16T23:30:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-16T23:30:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:54.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:30:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:30:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-16T23:30:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-16T23:30:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:30:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:30:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:30:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:30:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:30:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:30:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-16T23:30:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:30:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:30:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:30:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:30:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:30:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_37 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_37 2024-09-16T23:30:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_37.status 2024-09-16T23:30:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.status 2024-09-16T23:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-16T23:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_37.status 2024-09-16T23:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:30:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-16T23:30:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-16T23:30:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 158 2024-09-16T23:30:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.status 2024-09-16T23:30:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.status 2024-09-16T23:30:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-16T23:30:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:30:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=158 2024-09-16T23:30:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T23:30:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=71 2024-09-16T23:30:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 158 = 71 2024-09-16T23:30:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:30:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T23:30:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:30:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 158 = '' 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:30:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:30:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.status 2024-09-16T23:30:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-16T23:30:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:30:56.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=158 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 158 = 158 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:30:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:30:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_37': 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9f58b3639 2024-09-16T23:30:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9f58b3639 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:43 2024 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:43 2024 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:43 2024 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 33a3a258-5752-4a63-9f29-86e8aa61b524 2024-09-16T23:30:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:30:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-16T23:30:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.export 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.export 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.export 2024-09-16T23:30:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.export 2024-09-16T23:30:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.export 2024-09-16T23:30:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:30:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.export 2024-09-16T23:30:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_37.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_37.export 2024-09-16T23:30:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:30:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_38 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:30:58.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:30:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-16T23:30:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:30:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:00.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:00.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:00.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-16T23:31:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-16T23:31:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-16T23:31:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-16T23:31:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:00.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:00.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:00.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2024-09-16T23:31:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:00.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_38 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_38 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-16T23:31:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_38.status 2024-09-16T23:31:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.status 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_38.status 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-16T23:31:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 160 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.status 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.status 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-16T23:31:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:01.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=160 2024-09-16T23:31:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T23:31:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=72 2024-09-16T23:31:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 160 = 72 2024-09-16T23:31:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T23:31:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 160 = '' 2024-09-16T23:31:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:02.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.status 2024-09-16T23:31:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-16T23:31:02.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=160 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 160 = 160 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_38': 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9a86be00a 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9a86be00a 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:03.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:23:51 2024 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:23:51 2024 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:23:51 2024 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 94476c47-d77a-4fe0-9be9-c62c51a92c96 2024-09-16T23:31:03.829 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.export 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.export 2024-09-16T23:31:03.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.export 2024-09-16T23:31:03.833 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.export 2024-09-16T23:31:03.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:04.003 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.export 2024-09-16T23:31:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.export 2024-09-16T23:31:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_38.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_38.export 2024-09-16T23:31:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_39 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:05.396 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:31:05.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:31:05.397 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:31:05.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:31:05.401+0000 7fa41ecce640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 49108) UID: 1000 2024-09-16T23:31:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-16T23:31:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:05.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-16T23:31:05.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-16T23:31:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:05.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:05.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-16T23:31:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-16T23:31:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:05.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:05.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:05.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:05.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-16T23:31:05.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:05.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_39 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:05.766 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_39 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_39.status 2024-09-16T23:31:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.status 2024-09-16T23:31:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-16T23:31:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_39.status 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:05.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-16T23:31:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 162 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.status 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.status 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-16T23:31:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:06.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=162 2024-09-16T23:31:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T23:31:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 162 = '' 2024-09-16T23:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:07.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T23:31:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 162 = '' 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.status 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-16T23:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=162 2024-09-16T23:31:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 162 = 162 2024-09-16T23:31:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:08.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:09.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-16T23:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_39': 2024-09-16T23:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9fd0fc491 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9fd0fc491 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:04 2024 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:04 2024 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:04 2024 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 25336153-2db6-45d0-90fa-87fcf4dfdbf9 2024-09-16T23:31:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.export 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.export 2024-09-16T23:31:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.export 2024-09-16T23:31:09.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.export 2024-09-16T23:31:09.210 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.export 2024-09-16T23:31:09.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.export 2024-09-16T23:31:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_39.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_39.export 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_40 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:10.397 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:31:10.398 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:31:10.398 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:31:10.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:31:10.398 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:31:10.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:31:10.402 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2024-09-16T23:31:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:10.698 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:10.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-16T23:31:10.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-16T23:31:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:10.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:10.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-16T23:31:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-16T23:31:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-16T23:31:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:11.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_40 2024-09-16T23:31:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_40 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-16T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_40.status 2024-09-16T23:31:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.status 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_40.status 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-16T23:31:11.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 164 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.status 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.status 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-16T23:31:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=164 2024-09-16T23:31:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:11.762 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T23:31:11.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=74 2024-09-16T23:31:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 164 = 74 2024-09-16T23:31:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T23:31:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=74 2024-09-16T23:31:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 164 = 74 2024-09-16T23:31:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:13.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T23:31:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 164 = '' 2024-09-16T23:31:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:13.584 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.586+0000 7f821775c640 -1 reset not still connected to 0x55a23320c8f0 2024-09-16T23:31:13.587 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.588+0000 7f821775c640 -1 reset not still connected to 0x55a23320de10 2024-09-16T23:31:13.587 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.588+0000 7f821775c640 -1 reset not still connected to 0x55a2336da270 2024-09-16T23:31:13.587 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.588+0000 7f821775c640 -1 reset not still connected to 0x55a2336da410 2024-09-16T23:31:13.587 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.588+0000 7f821775c640 -1 reset not still connected to 0x55a2336dac30 2024-09-16T23:31:13.587 INFO:tasks.ceph.cluster2.osd.1.smithi098.stderr:2024-09-16T23:31:13.588+0000 7f821775c640 -1 reset not still connected to 0x55a2336db1e0 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.status 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-16T23:31:14.926 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:14.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=164 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 164 = 164 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:14.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_40': 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9e0627835 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9e0627835 2024-09-16T23:31:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:11 2024 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:11 2024 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:11 2024 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 709ec639-041d-469d-b611-2b988ea528b9 2024-09-16T23:31:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.export 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.export 2024-09-16T23:31:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.export 2024-09-16T23:31:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.export 2024-09-16T23:31:16.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.export 2024-09-16T23:31:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.export 2024-09-16T23:31:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_40.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_40.export 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_41 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:16.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-16T23:31:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:18.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:18.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-16T23:31:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T23:31:18.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-16T23:31:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:18.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-16T23:31:18.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-16T23:31:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:18.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:18.934 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-16T23:31:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_41 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_41 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_41.status 2024-09-16T23:31:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.status 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_41.status 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-16T23:31:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 166 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.status 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.status 2024-09-16T23:31:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-16T23:31:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=166 2024-09-16T23:31:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T23:31:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:20.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=75 2024-09-16T23:31:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 166 = 75 2024-09-16T23:31:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:20.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T23:31:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:20.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=75 2024-09-16T23:31:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 166 = 75 2024-09-16T23:31:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T23:31:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 166 = '' 2024-09-16T23:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.status 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-16T23:31:22.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=166 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 166 = 166 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-16T23:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_41': 2024-09-16T23:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:24.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c959f588ef 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c959f588ef 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:22 2024 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:22 2024 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:22 2024 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: c513aa23-d6df-4ccf-ba45-b813bb34e2b8 2024-09-16T23:31:24.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:24.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.export 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.export 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.export 2024-09-16T23:31:24.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.export 2024-09-16T23:31:24.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.export 2024-09-16T23:31:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.export 2024-09-16T23:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_41.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_41.export 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_42 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-16T23:31:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:24.732 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:24.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-16T23:31:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-16T23:31:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-16T23:31:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:25.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-16T23:31:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-16T23:31:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:25.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2024-09-16T23:31:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2024-09-16T23:31:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2024-09-16T23:31:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:26.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_42 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:26.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_42 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_42.status 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.status 2024-09-16T23:31:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-16T23:31:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_42.status 2024-09-16T23:31:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-16T23:31:26.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 168 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.status 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.status 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-16T23:31:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=168 2024-09-16T23:31:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T23:31:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=76 2024-09-16T23:31:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 168 = 76 2024-09-16T23:31:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T23:31:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=76 2024-09-16T23:31:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 168 = 76 2024-09-16T23:31:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T23:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:28.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 168 = '' 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:29.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.status 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-16T23:31:29.954 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=168 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 168 = 168 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:30.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_42': 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9d184007 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9d184007 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:32 2024 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:32 2024 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:32 2024 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 2a60c1ad-c75f-4e2d-a149-046ff2981c5f 2024-09-16T23:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.export 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.export 2024-09-16T23:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.export 2024-09-16T23:31:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.export 2024-09-16T23:31:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.export 2024-09-16T23:31:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.export 2024-09-16T23:31:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_42.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_42.export 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_43 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:31.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-16T23:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:31.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:31.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-16T23:31:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:32.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:32.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:32.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-16T23:31:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-16T23:31:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2024-09-16T23:31:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2024-09-16T23:31:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:32.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2024-09-16T23:31:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_43 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_43 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_43.status 2024-09-16T23:31:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.status 2024-09-16T23:31:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-16T23:31:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_43.status 2024-09-16T23:31:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-16T23:31:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 170 2024-09-16T23:31:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.status 2024-09-16T23:31:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.status 2024-09-16T23:31:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-16T23:31:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=170 2024-09-16T23:31:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:33.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T23:31:34.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=77 2024-09-16T23:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 170 = 77 2024-09-16T23:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:34.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T23:31:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=77 2024-09-16T23:31:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 170 = 77 2024-09-16T23:31:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:34.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T23:31:35.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 170 = '' 2024-09-16T23:31:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.status 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-16T23:31:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=170 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 170 = 170 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_43': 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c948606e8 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c948606e8 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:43 2024 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:43 2024 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:43 2024 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: f0c279cd-dbe9-4652-a6a2-a4416fd6df30 2024-09-16T23:31:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.export 2024-09-16T23:31:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.export 2024-09-16T23:31:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.export 2024-09-16T23:31:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.export 2024-09-16T23:31:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.export 2024-09-16T23:31:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.export 2024-09-16T23:31:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_43.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_43.export 2024-09-16T23:31:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_44 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-16T23:31:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:39.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-16T23:31:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T23:31:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:39.880 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:39.881 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:39.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:39.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:39.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-16T23:31:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-16T23:31:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:39.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:39.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:39.997 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2024-09-16T23:31:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_44 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_44 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_44.status 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.status 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-16T23:31:40.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_44.status 2024-09-16T23:31:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-16T23:31:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-16T23:31:41.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 172 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.status 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.status 2024-09-16T23:31:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-16T23:31:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=172 2024-09-16T23:31:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T23:31:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:41.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=78 2024-09-16T23:31:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 172 = 78 2024-09-16T23:31:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T23:31:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 172 = '' 2024-09-16T23:31:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T23:31:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:42.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:42.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 172 = '' 2024-09-16T23:31:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:43.585 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7891336859564132) > weight (0.25) 2024-09-16T23:31:43.585 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5399582282203403) > weight (0.25) 2024-09-16T23:31:43.585 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7289981277358201) > weight (0.25) 2024-09-16T23:31:43.585 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9066661353370769) > weight (0.25) 2024-09-16T23:31:43.585 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-16T23:31:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.status 2024-09-16T23:31:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-16T23:31:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:44.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=172 2024-09-16T23:31:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 172 = 172 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-16T23:31:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_44': 2024-09-16T23:31:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9db3e0208 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9db3e0208 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:24:52 2024 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:24:52 2024 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:24:52 2024 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 37e2b986-79ec-4fb7-9730-03a545b99092 2024-09-16T23:31:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-16T23:31:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.export 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.export 2024-09-16T23:31:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.export 2024-09-16T23:31:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.export 2024-09-16T23:31:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.export 2024-09-16T23:31:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.export 2024-09-16T23:31:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_44.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_44.export 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_45 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:46.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-16T23:31:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T23:31:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-16T23:31:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-16T23:31:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:47.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:47.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-16T23:31:47.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:47.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_45 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_45 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-16T23:31:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_45.status 2024-09-16T23:31:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.status 2024-09-16T23:31:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-16T23:31:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_45.status 2024-09-16T23:31:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-16T23:31:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-16T23:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 174 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.status 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.status 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-16T23:31:48.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=174 2024-09-16T23:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:48.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T23:31:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=79 2024-09-16T23:31:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 174 = 79 2024-09-16T23:31:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T23:31:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 174 = '' 2024-09-16T23:31:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T23:31:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:49.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 174 = '' 2024-09-16T23:31:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:49.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1.6 2024-09-16T23:31:51.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.status 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-16T23:31:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:51.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=174 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 174 = 174 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:51.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-16T23:31:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_45': 2024-09-16T23:31:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:52.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9cdba8847 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9cdba8847 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:06 2024 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:06 2024 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:06 2024 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:52.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: a03530bd-5bbe-4ed9-9353-0fe8900fb0a8 2024-09-16T23:31:52.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.export 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.export 2024-09-16T23:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.export 2024-09-16T23:31:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.export 2024-09-16T23:31:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.export 2024-09-16T23:31:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.export 2024-09-16T23:31:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_45.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_45.export 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_46 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:31:54.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:31:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:31:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:31:54.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:31:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:31:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-16T23:31:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:54.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:31:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:31:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:31:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:31:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-16T23:31:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T23:31:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:31:54.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:31:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:31:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:31:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2024-09-16T23:31:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2024-09-16T23:31:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:31:54.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:31:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:31:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:31:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:31:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:31:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:31:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:31:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2024-09-16T23:31:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:31:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:31:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:31:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:31:54.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_46 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_46 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_46.status 2024-09-16T23:31:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.status 2024-09-16T23:31:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-16T23:31:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_46.status 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-16T23:31:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 176 2024-09-16T23:31:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.status 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.status 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-16T23:31:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:31:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=176 2024-09-16T23:31:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T23:31:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 176 = '' 2024-09-16T23:31:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T23:31:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:31:55.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 176 = '' 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:31:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:31:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.status 2024-09-16T23:31:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-16T23:31:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:31:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=176 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 176 = 176 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:31:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:31:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_46': 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9b26b3831 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9b26b3831 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:31:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:18 2024 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:18 2024 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:18 2024 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: ae447345-839d-464e-9a21-3546356877ae 2024-09-16T23:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:31:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.export 2024-09-16T23:31:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.export 2024-09-16T23:31:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.export 2024-09-16T23:31:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.export 2024-09-16T23:31:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:57.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.export 2024-09-16T23:31:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:31:58.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.export 2024-09-16T23:31:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_46.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_46.export 2024-09-16T23:31:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:31:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:31:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_47 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:31:58.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:31:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:31:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:31:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:31:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:32:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:32:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:32:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:32:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:32:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-16T23:32:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:32:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:32:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:32:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:32:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:32:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:32:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:32:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-16T23:32:00.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-16T23:32:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:00.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:32:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:32:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:32:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-16T23:32:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-16T23:32:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:32:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:32:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:32:00.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:32:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-16T23:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:32:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:32:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_47 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:32:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_47 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_47.status 2024-09-16T23:32:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.status 2024-09-16T23:32:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-16T23:32:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_47.status 2024-09-16T23:32:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:00.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-16T23:32:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 178 2024-09-16T23:32:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.status 2024-09-16T23:32:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.status 2024-09-16T23:32:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-16T23:32:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=178 2024-09-16T23:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:32:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T23:32:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=81 2024-09-16T23:32:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 178 = 81 2024-09-16T23:32:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:32:01.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T23:32:01.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:32:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:32:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:32:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:32:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 178 = '' 2024-09-16T23:32:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:32:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.status 2024-09-16T23:32:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-16T23:32:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:02.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=178 2024-09-16T23:32:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 178 = 178 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:32:02.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_47': 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c9dfbe3b61 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c9dfbe3b61 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:32:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:25 2024 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:25 2024 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:25 2024 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 52b8a006-6595-4b78-a489-5a8e09247cf9 2024-09-16T23:32:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.export 2024-09-16T23:32:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.export 2024-09-16T23:32:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.export 2024-09-16T23:32:03.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.export 2024-09-16T23:32:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.export 2024-09-16T23:32:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:04.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.export 2024-09-16T23:32:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_47.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_47.export 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_48 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:32:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:32:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:32:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-16T23:32:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:32:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:32:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:32:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:32:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:32:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:32:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-16T23:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:32:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:32:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:32:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:32:06.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:32:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:32:06.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:32:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-16T23:32:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-16T23:32:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:32:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:06.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:32:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:32:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-16T23:32:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-16T23:32:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:06.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-16T23:32:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:32:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:06.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:32:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:32:06.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:32:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-16T23:32:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:32:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:32:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:32:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:32:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:32:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_48 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_48 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:06.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_48.status 2024-09-16T23:32:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.status 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_48.status 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-16T23:32:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 180 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.status 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.status 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-16T23:32:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:32:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=180 2024-09-16T23:32:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:32:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:07.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T23:32:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:07.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:07.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=82 2024-09-16T23:32:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 180 = 82 2024-09-16T23:32:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:07.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T23:32:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:07.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:32:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:32:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:32:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:32:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 180 = '' 2024-09-16T23:32:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.status 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-16T23:32:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=180 2024-09-16T23:32:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 180 = 180 2024-09-16T23:32:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:09.567 INFO:tasks.ceph.cluster1.osd.1.smithi096.stderr:2024-09-16T23:32:09.569+0000 7f020bb14640 -1 reset not still connected to 0x55b17a6cc680 2024-09-16T23:32:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_48': 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c918b833f0 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c918b833f0 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:32:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:34 2024 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:34 2024 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:34 2024 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 31266f8d-efc2-46a5-bb23-092b5e031f3a 2024-09-16T23:32:09.775 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:32:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.export 2024-09-16T23:32:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.export 2024-09-16T23:32:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.export 2024-09-16T23:32:09.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.export 2024-09-16T23:32:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.export 2024-09-16T23:32:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.export 2024-09-16T23:32:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_48.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_48.export 2024-09-16T23:32:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_49 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:32:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:32:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:32:12.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:32:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:32:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:32:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-16T23:32:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T23:32:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:32:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:32:12.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:32:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-16T23:32:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-16T23:32:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:32:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:32:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:12.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-16T23:32:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:32:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:32:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:32:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:32:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:32:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:32:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_49 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_49 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-16T23:32:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_49.status 2024-09-16T23:32:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.status 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_49.status 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-16T23:32:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 182 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.status 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.status 2024-09-16T23:32:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-16T23:32:13.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:32:13.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=182 2024-09-16T23:32:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T23:32:13.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=83 2024-09-16T23:32:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 182 = 83 2024-09-16T23:32:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T23:32:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:32:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 182 = '' 2024-09-16T23:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.status 2024-09-16T23:32:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-16T23:32:14.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:14.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=182 2024-09-16T23:32:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 182 = 182 2024-09-16T23:32:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-16T23:32:14.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:32:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_49': 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c92ca146ac 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c92ca146ac 2024-09-16T23:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:43 2024 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:43 2024 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:43 2024 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: aaa3557b-c9d4-42bc-827a-961b199be7de 2024-09-16T23:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.export 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.export 2024-09-16T23:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.export 2024-09-16T23:32:15.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.export 2024-09-16T23:32:15.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.export 2024-09-16T23:32:16.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.export 2024-09-16T23:32:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_49.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_49.export 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_50 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=started 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=started 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_result 2024-09-16T23:32:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=stopped 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_instance=cluster1 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -i state 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance=cluster1 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local loop_instance 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sleep 0 2024-09-16T23:32:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-16T23:32:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ seq 0 3 2024-09-16T23:32:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:0 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:0 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=0 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:0 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:0 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=0 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=0 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=0 2024-09-16T23:32:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T23:32:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T23:32:18.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-16T23:32:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-16T23:32:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:1 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:1 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-16T23:32:18.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=1 2024-09-16T23:32:18.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-16T23:32:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-16T23:32:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-16T23:32:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-16T23:32:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd mirror status : get status for rbd mirror 2024-09-16T23:32:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:admin_socket: invalid command 2024-09-16T23:32:18.334 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1:2 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ shift 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local val=cluster1:2 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster_var_name=cluster 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local instance_var_name=instance 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=2 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval cluster=cluster1 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ eval instance=2 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local local_cluster=cluster1:2 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster=cluster1 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local val=cluster1:2 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local cluster_var_name=local_cluster 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ local instance_var_name=instance 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ cluster=cluster1 2024-09-16T23:32:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ instance=2 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval local_cluster=cluster1 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ local_cluster=cluster1 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ eval instance=2 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ instance=2 2024-09-16T23:32:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-16T23:32:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-16T23:32:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-16T23:32:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ return 0 2024-09-16T23:32:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ status_result=' "state": "Replaying"' 2024-09-16T23:32:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo ' "state": "Replaying"' 2024-09-16T23:32:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ grep -i Replaying 2024-09-16T23:32:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: "state": "Replaying" 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=started 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test started = started 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_50 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = journal ']' 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' snapshot = snapshot ']' 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_50 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_cluster=cluster1 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_pool=mirror 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local remote_pool=mirror 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster2-mirror-image_50.status 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local status_log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.status 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-16T23:32:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ echo cluster1-mirror-image_50.status 2024-09-16T23:32:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-16T23:32:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local local_status_log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-16T23:32:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-16T23:32:19.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Snapshot ID: 184 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.status 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.status 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-16T23:32:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-16T23:32:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snapshot_id=184 2024-09-16T23:32:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.2 2024-09-16T23:32:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:19.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T23:32:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=84 2024-09-16T23:32:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 184 = 84 2024-09-16T23:32:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.4 2024-09-16T23:32:19.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T23:32:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:32:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:-:1.1: Document is empty 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:^ 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id= 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 184 = '' 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-16T23:32:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.8 2024-09-16T23:32:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local log=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.status 2024-09-16T23:32:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-16T23:32:20.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-16T23:32:20.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-16T23:32:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local primary_snapshot_id=184 2024-09-16T23:32:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 184 = 184 2024-09-16T23:32:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-16T23:32:20.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:32:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:20.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:32:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-16T23:32:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:32:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-16T23:32:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_50': 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 128 MiB in 32 objects 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 2 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 10c91901f202 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.10c91901f202 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering, non-primary 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:25:57 2024 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:25:57 2024 2024-09-16T23:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:25:57 2024 2024-09-16T23:32:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: protected: False 2024-09-16T23:32:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:32:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:32:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b056f8d2-1f2f-4deb-8a45-c78bb6111629 2024-09-16T23:32:21.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: false 2024-09-16T23:32:21.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap_name=snap 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local ret=0 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local rmt_export=/tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.export 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local loc_export=/tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.export 2024-09-16T23:32:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.export 2024-09-16T23:32:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.export 2024-09-16T23:32:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.export 2024-09-16T23:32:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-16T23:32:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cmp /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.export 2024-09-16T23:32:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -f /tmp/tmp.JxTgO6em7N/cluster2-mirror-image_50.export /tmp/tmp.JxTgO6em7N/cluster1-mirror-image_50.export 2024-09-16T23:32:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: delete many images' 2024-09-16T23:32:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:32:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:32:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:32:22 TEST: delete many images 2024-09-16T23:32:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:32:22 TEST: delete many images 2024-09-16T23:32:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 50 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_1 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-16T23:32:23.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:32:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:23.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:32:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:32:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:32:23.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:23.296+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca843d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:23.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:23.296+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0daf00 [3/2fda3e9f-ae26-4e8f-b0fd-5a14428de02e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:23.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:23.597+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6c9964340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:23.595 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:23.597+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0daf00 [3/2fda3e9f-ae26-4e8f-b0fd-5a14428de02e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:24.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_2 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-16T23:32:24.619 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:32:24.621+0000 7f8c0afcb640 -1 reset not still connected to 0x558f15c46f70 2024-09-16T23:32:24.620 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:32:24.621+0000 7f8c0afcb640 -1 reset not still connected to 0x558f195f4750 2024-09-16T23:32:24.620 INFO:tasks.ceph.cluster1.osd.0.smithi096.stderr:2024-09-16T23:32:24.621+0000 7f8c0afcb640 -1 reset not still connected to 0x558f1a1feea0 2024-09-16T23:32:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:25.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:32:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:32:25.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:32:25.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:25.317+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:25.314 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:25.317+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0db400 [3/72e6208d-3742-4bfa-8918-249d11759c7e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_3 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-16T23:32:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:27.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:32:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:32:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:32:27.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:27.322+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca80f040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:27.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:27.322+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810280 [3/7dc49f0a-1a35-4bab-aea4-16de6b2f77bc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_4 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-16T23:32:29.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:32:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:32:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:32:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:32:29.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:29.323+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_5 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-16T23:32:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:32:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:32:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:32:31.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:31.324+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca80e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:31.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:31.324+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca876500 [3/f84d3a0f-2e8c-472f-b6aa-47f1ebb35940] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_6 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:32:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-16T23:32:32.402 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.29861734220678937) > weight (0.25) 2024-09-16T23:32:32.403 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T23:32:32.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T23:32:32.403 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-16T23:32:32.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:32:32.407+0000 7fa147b14640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 49947) UID: 1000 2024-09-16T23:32:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:32:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:32:33.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:32:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:32:33.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:33.339+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:33.336 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:33.339+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83c780 [3/c94c7221-b8a2-4e38-a50c-e34a1f45df1c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:33.404 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T23:32:33.405 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:32:33.405 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T23:32:33.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T23:32:33.406 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T23:32:33.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T23:32:33.408 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-16T23:32:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_7 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:32:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-16T23:32:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:32:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:32:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:32:35.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:32:35.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:35.222+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca80eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:35.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:35.222+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f7900 [3/8e4da14b-dfc0-4158-b26e-7af55852456e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:36.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_8 2024-09-16T23:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-16T23:32:36.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:32:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-16T23:32:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:32:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:32:37.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:32:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:32:37.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:37.349+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca80fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:37.346 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:37.349+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83c000 [3/bd37f003-9696-4966-8734-3bc10187a7c9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_9 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:38.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-16T23:32:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:32:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:32:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:32:39.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:39.351+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:39.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:39.351+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f7400 [3/89e925c9-d5c8-42fb-9de9-72a27fc3d8b5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_10 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-16T23:32:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:32:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:32:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:32:41.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:32:41.569 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:32:41.571+0000 7f440f149640 -1 reset not still connected to 0x55f27eb2b2b0 2024-09-16T23:32:41.569 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:32:41.571+0000 7f440f149640 -1 reset not still connected to 0x55f27f53eea0 2024-09-16T23:32:41.570 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:32:41.572+0000 7f440f149640 -1 reset not still connected to 0x55f291dccf70 2024-09-16T23:32:41.570 INFO:tasks.ceph.cluster2.osd.0.smithi098.stderr:2024-09-16T23:32:41.572+0000 7f440f149640 -1 reset not still connected to 0x55f293e78f70 2024-09-16T23:32:41.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:41.575+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:41.577 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:41.579+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83cf00 [3/ec61c977-0c0f-41e7-8a23-423f6faad7dc] start: already running 2024-09-16T23:32:41.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:41.581+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:41.579 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:41.581+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83cf00 [3/ec61c977-0c0f-41e7-8a23-423f6faad7dc] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:42.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_11 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:32:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-16T23:32:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_11 2024-09-16T23:32:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:32:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-16T23:32:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-16T23:32:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:43.384+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:43.382 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:43.384+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0daa00 [3/268e0ae9-2b9e-4a82-8b83-58d27f2a3cd5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_12 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:44.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-16T23:32:45.069 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-16T23:32:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:32:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_12 2024-09-16T23:32:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:32:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-16T23:32:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-16T23:32:45.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:45.233+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:45.231 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:45.233+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810a00 [3/b9122161-1c25-422e-ae7b-083efda568a9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_13 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:46.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:32:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:46.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-16T23:32:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_13 2024-09-16T23:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-16T23:32:47.289 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-16T23:32:47.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:47.387+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:47.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:47.387+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca811b80 [3/b92b2830-e993-4cb1-800e-5126fdb3887f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_14 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-16T23:32:49.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:49.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-16T23:32:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:32:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_14 2024-09-16T23:32:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:32:49.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-16T23:32:49.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-16T23:32:49.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:49.401+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:49.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:49.401+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83ca00 [3/ca2b9bfb-9340-437f-9dcd-274b942d579d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:49.586 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.29368785876470416) > weight (0.25) 2024-09-16T23:32:49.587 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5791535511367596) > weight (0.25) 2024-09-16T23:32:49.587 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-16T23:32:49.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-16T23:32:49.587 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-16T23:32:49.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:32:49.590+0000 7f905f5a9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 43213) UID: 1000 2024-09-16T23:32:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_15 2024-09-16T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-16T23:32:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-16T23:32:51.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:51.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-16T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:51.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_15 2024-09-16T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:32:51.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-16T23:32:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-16T23:32:51.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:32:51.374+0000 7f3e137b1640 -1 rbd::mirror::ImageMap: 0x556522096c60 handle_peer_ack_remove: failed to remove global_image_id=d611b2eb-d352-4abb-977d-a94f77c573c9 2024-09-16T23:32:52.355 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_16 2024-09-16T23:32:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-16T23:32:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:32:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-16T23:32:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_16 2024-09-16T23:32:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:32:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-16T23:32:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-16T23:32:53.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:53.593+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:53.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:53.593+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0da500 [3/161cd388-c85e-48c4-8c8e-84aaca587105] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:53.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:53.737+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:53.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:53.737+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0da500 [3/161cd388-c85e-48c4-8c8e-84aaca587105] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_17 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:32:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:54.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-16T23:32:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_17 2024-09-16T23:32:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:32:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-16T23:32:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-16T23:32:55.228 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:55.230+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca901a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:55.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:55.230+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6f00 [3/77e8cc56-9128-4126-9b3a-02b903130874] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_18 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:32:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-16T23:32:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-16T23:32:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:32:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_18 2024-09-16T23:32:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:32:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-16T23:32:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-16T23:32:57.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:57.235+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9001a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:32:57.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:57.235+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6c80 [3/58af7715-4fc6-402e-aa27-d73e969a9d7a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:32:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:32:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:32:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:32:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_19 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:32:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-16T23:32:58.588 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-16T23:32:58.589 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:32:58.589 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-16T23:32:58.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-16T23:32:58.589 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-16T23:32:58.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-16T23:32:58.592 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-16T23:32:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:32:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_19 2024-09-16T23:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:32:59.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:32:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:32:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-16T23:32:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-16T23:32:59.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:32:59.234+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:00.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:00.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_20 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-16T23:33:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_20 2024-09-16T23:33:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:33:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-16T23:33:01.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-16T23:33:01.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:01.255+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9016c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:01.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:01.255+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6500 [3/3e27fd33-23b8-4a4b-a8ee-9baf2df8b8f0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:02.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:02.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_21 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:02.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-16T23:33:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-16T23:33:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:33:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_21 2024-09-16T23:33:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:33:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-16T23:33:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-16T23:33:03.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:03.250+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:03.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:03.250+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca876000 [3/eecd37c8-2507-4ea3-8c85-72ea35b897c6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_22 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-16T23:33:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_22 2024-09-16T23:33:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:33:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-16T23:33:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-16T23:33:05.232 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:05.234+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca842d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:05.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:05.234+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca876780 [3/fb737a24-defd-44de-8405-d2f600fcfa9d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_23 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-16T23:33:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-16T23:33:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:07.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:33:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_23 2024-09-16T23:33:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:33:07.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-16T23:33:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-16T23:33:07.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:07.275+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca842000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:07.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:07.275+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83d400 [3/e1d61309-9267-41b0-b201-a6011fded361] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_24 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-16T23:33:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_24 2024-09-16T23:33:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:33:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-16T23:33:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-16T23:33:09.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:09.287+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:09.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:09.287+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6780 [3/449d20a2-3295-43fc-99a8-bae9ff72cca1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_25 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:33:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-16T23:33:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-16T23:33:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:33:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_25 2024-09-16T23:33:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:33:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-16T23:33:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-16T23:33:11.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:11.277+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:11.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:11.277+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca811400 [3/aa545518-746e-469d-a87e-2348440b9692] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:12.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_26 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:12.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-16T23:33:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-16T23:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_26 2024-09-16T23:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:33:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-16T23:33:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-16T23:33:13.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:13.386+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:13.385 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:13.387+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6280 [3/3a054c79-09d1-4ef0-b9c5-083578f40968] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_27 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-16T23:33:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_27 2024-09-16T23:33:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:33:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-16T23:33:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-16T23:33:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:15.444+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:15.442 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:15.444+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0da000 [3/0de0d1ea-4ea9-47eb-a8fc-cc4b4c55433f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:16.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_28 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:16.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-16T23:33:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:17.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-16T23:33:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:33:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:17.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_28 2024-09-16T23:33:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:17.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:33:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-16T23:33:17.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-16T23:33:17.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:17.329+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:17.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:17.329+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810000 [3/77ea12b7-17e8-4b4a-ad76-3f55bd6a6279] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_29 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:33:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-16T23:33:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-16T23:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_29 2024-09-16T23:33:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:33:19.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-16T23:33:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-16T23:33:19.529 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:19.531+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:19.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:19.532+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca811900 [3/d119fabc-3461-4f14-ad5b-937b0502300c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_30 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-16T23:33:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_30 2024-09-16T23:33:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:33:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-16T23:33:21.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-16T23:33:21.343 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:21.345+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:21.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:21.345+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810c80 [3/9a9a2e05-0891-4a2c-bb16-07ba21d8071f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:22.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_31 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-16T23:33:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_31 2024-09-16T23:33:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:33:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-16T23:33:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-16T23:33:23.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:23.352+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:23.351 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:23.352+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83c500 [3/c6104db7-e7bf-4eea-a889-773f358930c7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:23.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:23.592+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca842b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:23.591 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:23.592+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83c500 [3/c6104db7-e7bf-4eea-a889-773f358930c7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:24.203 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.205+0000 7fbcef505640 -1 reset not still connected to 0x565180d9fba0 2024-09-16T23:33:24.208 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.208+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a6750 2024-09-16T23:33:24.208 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.208+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a7e10 2024-09-16T23:33:24.208 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.208+0000 7fbcef505640 -1 reset not still connected to 0x565199734000 2024-09-16T23:33:24.208 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.208+0000 7fbcef505640 -1 reset not still connected to 0x5651a0ce7ee0 2024-09-16T23:33:24.208 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:24.208+0000 7fbcef505640 -1 reset not still connected to 0x5651a17635f0 2024-09-16T23:33:24.300 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_32 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-16T23:33:25.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_32 2024-09-16T23:33:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:33:25.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-16T23:33:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-16T23:33:25.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:25.239+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9004e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:25.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:25.239+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0da280 [3/11ebbc36-f469-465c-b15a-fe40b29147fb] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_33 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:26.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-16T23:33:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-16T23:33:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:33:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_33 2024-09-16T23:33:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:33:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-16T23:33:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-16T23:33:27.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:27.358+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:27.356 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:27.358+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83cc80 [3/e491a9c0-f499-472f-b800-fabf601f9b18] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_34 2024-09-16T23:33:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-16T23:33:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:33:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-16T23:33:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:29.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_34 2024-09-16T23:33:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:33:29.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-16T23:33:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-16T23:33:29.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:29.360+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:29.359 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:29.360+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0db680 [3/2dc3b71d-4f9f-4123-bb64-1b41606c25b5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_35 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:33:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:30.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-16T23:33:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_35 2024-09-16T23:33:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:33:31.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-16T23:33:31.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-16T23:33:31.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:31.377+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca8436c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:31.375 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:31.377+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca811180 [3/a2742cc4-6203-4618-aa8e-430f3fe6d0e9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_36 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:33:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-16T23:33:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_36 2024-09-16T23:33:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:33:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-16T23:33:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-16T23:33:33.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:33.380+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:33.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:33.380+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6c99e6000 [3/03dea910-3ab0-46f2-a90e-2ddda005fc50] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:34.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_37 2024-09-16T23:33:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-16T23:33:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:33:34.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-16T23:33:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-16T23:33:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:33:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_37 2024-09-16T23:33:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:33:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-16T23:33:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-16T23:33:35.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:35.246+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:35.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:35.246+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0dbb80 [3/33a3a258-5752-4a63-9f29-86e8aa61b524] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_38 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:36.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-16T23:33:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_38 2024-09-16T23:33:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:33:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-16T23:33:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-16T23:33:37.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:37.391+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca901380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:37.388 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:37.391+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f7b80 [3/94476c47-d77a-4fe0-9be9-c62c51a92c96] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:38.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_39 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:38.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:33:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:38.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-16T23:33:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:39.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_39 2024-09-16T23:33:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:33:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-16T23:33:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-16T23:33:39.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:39.409+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9011e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:39.407 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:39.409+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0da780 [3/25336153-2db6-45d0-90fa-87fcf4dfdbf9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_40 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:40.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:33:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-16T23:33:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:41.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_40 2024-09-16T23:33:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:33:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-16T23:33:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-16T23:33:41.406 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:41.409+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9009c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:42.276 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56517cf78a90 2024-09-16T23:33:42.276 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56517e64b2b0 2024-09-16T23:33:42.276 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56517e64b380 2024-09-16T23:33:42.276 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56517f4fbe10 2024-09-16T23:33:42.277 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x565180d9fba0 2024-09-16T23:33:42.277 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a6750 2024-09-16T23:33:42.277 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x56518d7a7e10 2024-09-16T23:33:42.277 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x5651a0ce7ee0 2024-09-16T23:33:42.277 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:33:42.278+0000 7fbcef505640 -1 reset not still connected to 0x5651a17635f0 2024-09-16T23:33:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_41 2024-09-16T23:33:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-16T23:33:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:33:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-16T23:33:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-16T23:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_41 2024-09-16T23:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-16T23:33:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-16T23:33:43.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:43.419+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca901d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:43.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:43.419+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca83c280 [3/c513aa23-d6df-4ccf-ba45-b813bb34e2b8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_42 2024-09-16T23:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-16T23:33:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:33:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:44.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-16T23:33:45.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-16T23:33:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:33:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:45.088 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_42 2024-09-16T23:33:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:33:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-16T23:33:45.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-16T23:33:45.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:45.240+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:45.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:45.240+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0dac80 [3/2a60c1ad-c75f-4e2d-a149-046ff2981c5f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_43 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:46.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:33:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-16T23:33:47.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_43 2024-09-16T23:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:33:47.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-16T23:33:47.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-16T23:33:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:47.249+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca842340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:47.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:47.249+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca876280 [3/f0c279cd-dbe9-4652-a6a2-a4416fd6df30] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_44 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-16T23:33:49.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:49.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_44 2024-09-16T23:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:33:49.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-16T23:33:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-16T23:33:49.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:49.252+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca9004e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:49.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:49.252+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6000 [3/37e2b986-79ec-4fb7-9730-03a545b99092] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:50.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_45 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-16T23:33:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-16T23:33:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:33:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_45 2024-09-16T23:33:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:33:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-16T23:33:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-16T23:33:51.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:51.253+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca8421a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:51.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:51.253+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810f00 [3/a03530bd-5bbe-4ed9-9353-0fe8900fb0a8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_46 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-16T23:33:53.099 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-16T23:33:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:33:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:53.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_46 2024-09-16T23:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:33:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-16T23:33:53.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-16T23:33:53.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:53.260+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:53.258 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:53.260+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810500 [3/ae447345-839d-464e-9a21-3546356877ae] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:53.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:53.586+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:53.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:53.586+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810500 [3/ae447345-839d-464e-9a21-3546356877ae] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_47 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-16T23:33:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_47 2024-09-16T23:33:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:33:55.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-16T23:33:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-16T23:33:55.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:55.243+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:55.240 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:55.243+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6a00 [3/52b8a006-6595-4b78-a489-5a8e09247cf9] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_48 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-16T23:33:57.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:57.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_48 2024-09-16T23:33:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:33:57.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-16T23:33:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-16T23:33:57.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:57.290+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:33:57.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:57.290+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0db900 [3/31266f8d-efc2-46a5-bb23-092b5e031f3a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:33:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_49 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:33:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-16T23:33:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:33:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-16T23:33:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:33:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:33:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:33:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_49 2024-09-16T23:33:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:33:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:33:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:33:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-16T23:33:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-16T23:33:59.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:33:59.289+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca8429c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:34:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_50 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:00.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local snap=snap 2024-09-16T23:34:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-16T23:34:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing snap: 100% complete...done. 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:01.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:34:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_50 2024-09-16T23:34:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:34:01.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-16T23:34:01.189 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-16T23:34:01.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:34:01.502+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca843380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:34:01.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:34:01.502+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810780 [3/b056f8d2-1f2f-4deb-8a45-c78bb6111629] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:34:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-16T23:34:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-16T23:34:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:34:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:34:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:34:02 TEST: image deletions should propagate 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:34:02 TEST: image deletions should propagate 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_count=0 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local count 2024-09-16T23:34:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local last_count=0 2024-09-16T23:34:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ true 2024-09-16T23:34:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:34:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:34:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T23:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'images: ' 2024-09-16T23:34:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 2 2024-09-16T23:34:02.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:34:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ count=0 2024-09-16T23:34:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:34:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state 2024-09-16T23:34:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 40 2024-09-16T23:34:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in `seq 1 40` 2024-09-16T23:34:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 1 -ne 1 2024-09-16T23:34:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-16T23:34:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep 'image health:' 2024-09-16T23:34:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cut '-d ' -f 3 2024-09-16T23:34:02.409 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8488049500879316) > weight (0.25) 2024-09-16T23:34:02.410 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-16T23:34:02.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-16T23:34:02.412 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-16T23:34:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: failed to query services: (13) Permission denied 2024-09-16T23:34:02.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:34:02.413+0000 7f9e3d8ce640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 63034) UID: 1000 2024-09-16T23:34:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ state=OK 2024-09-16T23:34:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = ERROR 2024-09-16T23:34:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test OK = OK 2024-09-16T23:34:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 50 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_1 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.418 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T23:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.563 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:02.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_2 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:34:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T23:34:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T23:34:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:34:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_3 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:34:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T23:34:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:02.838 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T23:34:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:34:02.940 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:34:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:02.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:02.980 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_4 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:34:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T23:34:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:03.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T23:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_5 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:34:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:34:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T23:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:34:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:34:03.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_6 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:34:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T23:34:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:34:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_7 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:34:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:34:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T23:34:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:03.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:34:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_8 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T23:34:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:34:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T23:34:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T23:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:34:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:34:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_9 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:34:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T23:34:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:34:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T23:34:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_10 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:34:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:34:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T23:34:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:34:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:34:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_11 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_11 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T23:34:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T23:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_11 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.583 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_11 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_11 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-16T23:34:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-16T23:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_12 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:34:04.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_12 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T23:34:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T23:34:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_12 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_12 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_12 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-16T23:34:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_13 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.804 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_13 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T23:34:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T23:34:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_13 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_13 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_13 2024-09-16T23:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-16T23:34:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:04.982 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_14 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_14 2024-09-16T23:34:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T23:34:04.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T23:34:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_14 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:05.122 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_14 2024-09-16T23:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_14 2024-09-16T23:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-16T23:34:05.123 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-16T23:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_15 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_15 2024-09-16T23:34:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T23:34:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T23:34:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_15 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_15 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_15 2024-09-16T23:34:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-16T23:34:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-16T23:34:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_16 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_16 2024-09-16T23:34:05.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T23:34:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T23:34:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_16 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:05.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_16 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_16 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-16T23:34:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-16T23:34:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_17 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T23:34:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_17 2024-09-16T23:34:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T23:34:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T23:34:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-16T23:34:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_17 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_17 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_17 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-16T23:34:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-16T23:34:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_18 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T23:34:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_18 2024-09-16T23:34:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T23:34:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T23:34:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_18 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_18 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_18 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-16T23:34:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-16T23:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_19 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_19 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T23:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T23:34:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_19 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_19 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_19 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-16T23:34:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_20 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_20 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T23:34:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T23:34:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_20 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_20 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_20 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-16T23:34:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_21 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-16T23:34:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_21 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T23:34:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T23:34:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:06.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_21 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_21 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.386 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_21 2024-09-16T23:34:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-16T23:34:06.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-16T23:34:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_22 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_22 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T23:34:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T23:34:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:06.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_22 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.565 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_22 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_22 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-16T23:34:06.566 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-16T23:34:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_23 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:34:06.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_23 2024-09-16T23:34:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T23:34:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T23:34:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_23 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_23 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_23 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-16T23:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_24 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_24 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T23:34:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T23:34:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_24 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_24 2024-09-16T23:34:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_24 2024-09-16T23:34:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-16T23:34:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-16T23:34:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_25 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T23:34:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_25 2024-09-16T23:34:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T23:34:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T23:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:07.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_25 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_25 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_25 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-16T23:34:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-16T23:34:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_26 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_26 2024-09-16T23:34:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T23:34:07.152 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T23:34:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_26 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_26 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_26 2024-09-16T23:34:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-16T23:34:07.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-16T23:34:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:07.330 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_27 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_27 2024-09-16T23:34:07.331 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T23:34:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T23:34:07.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_27 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_27 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_27 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-16T23:34:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-16T23:34:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_28 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_28 2024-09-16T23:34:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T23:34:07.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T23:34:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-16T23:34:07.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_28 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_28 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_28 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-16T23:34:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-16T23:34:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_29 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T23:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_29 2024-09-16T23:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T23:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T23:34:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-16T23:34:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_29 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_29 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_29 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-16T23:34:07.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_30 2024-09-16T23:34:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_30 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T23:34:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T23:34:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_30 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_30 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_30 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-16T23:34:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_31 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_31 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T23:34:08.057 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T23:34:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_31 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_31 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_31 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-16T23:34:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-16T23:34:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_32 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_32 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T23:34:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T23:34:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_32 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_32 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_32 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-16T23:34:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_33 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:34:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_33 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T23:34:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T23:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.457 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-16T23:34:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_33 2024-09-16T23:34:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_33 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_33 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-16T23:34:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-16T23:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_34 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T23:34:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_34 2024-09-16T23:34:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T23:34:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T23:34:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_34 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_34 2024-09-16T23:34:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_34 2024-09-16T23:34:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-16T23:34:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-16T23:34:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_35 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_35 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T23:34:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T23:34:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-16T23:34:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_35 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_35 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_35 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-16T23:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-16T23:34:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_36 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_36 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T23:34:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T23:34:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-16T23:34:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_36 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_36 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_36 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-16T23:34:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_37 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:09.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_37 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T23:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T23:34:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_37 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:09.280 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_37 2024-09-16T23:34:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_37 2024-09-16T23:34:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-16T23:34:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-16T23:34:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_38 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T23:34:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_38 2024-09-16T23:34:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T23:34:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T23:34:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_38 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_38 2024-09-16T23:34:09.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_38 2024-09-16T23:34:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-16T23:34:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_39 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_39 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:09.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_39 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_39 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_39 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-16T23:34:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:09.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_40 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_40 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T23:34:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T23:34:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:09.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-16T23:34:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_40 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_40 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_40 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-16T23:34:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-16T23:34:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_41 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T23:34:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_41 2024-09-16T23:34:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T23:34:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T23:34:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_41 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_41 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_41 2024-09-16T23:34:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-16T23:34:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-16T23:34:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_42 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_42 2024-09-16T23:34:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T23:34:10.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T23:34:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_42 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_42 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_42 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-16T23:34:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_43 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:34:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_43 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T23:34:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T23:34:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-16T23:34:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_43 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_43 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_43 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-16T23:34:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-16T23:34:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_44 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T23:34:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_44 2024-09-16T23:34:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T23:34:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.413 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-16T23:34:10.413 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:34:10.414 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-16T23:34:10.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-16T23:34:10.414 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-16T23:34:10.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-16T23:34:10.417 INFO:tasks.rbd_mirror_thrash:waiting for 61 secs before thrashing 2024-09-16T23:34:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T23:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_44 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_44 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_44 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-16T23:34:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_45 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_45 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T23:34:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.593 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5065338201628068) > weight (0.25) 2024-09-16T23:34:10.593 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3905573975243425) > weight (0.25) 2024-09-16T23:34:10.593 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.922244721493111) > weight (0.25) 2024-09-16T23:34:10.593 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6660324120623018) > weight (0.25) 2024-09-16T23:34:10.593 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2024-09-16T23:34:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T23:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-16T23:34:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_45 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_45 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_45 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-16T23:34:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_46 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_46 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T23:34:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T23:34:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_46 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_46 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_46 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-16T23:34:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_47 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_47 2024-09-16T23:34:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T23:34:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:10.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T23:34:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-16T23:34:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_47 2024-09-16T23:34:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_47 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_47 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-16T23:34:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_48 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:34:11.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_48 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T23:34:11.130 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T23:34:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-16T23:34:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_48 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_48 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_48 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-16T23:34:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_49 2024-09-16T23:34:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_49 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T23:34:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T23:34:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_49 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_49 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_49 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-16T23:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_50 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_50 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T23:34:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.522 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T23:34:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-16T23:34:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_50 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_50 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_50 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-16T23:34:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:34:11.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:34:11 TEST: delete images during bootstrap 2024-09-16T23:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:34:11 TEST: delete images during bootstrap 2024-09-16T23:34:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-16T23:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=image 2024-09-16T23:34:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-16T23:34:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd: mirroring is already configured for image mode 2024-09-16T23:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-16T23:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=image 2024-09-16T23:34:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-16T23:34:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd: mirroring is already configured for image mode 2024-09-16T23:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ start_mirror cluster1 2024-09-16T23:34:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local val=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster_var_name=cluster 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local instance_var_name=instance 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cluster=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ instance=0 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval cluster=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ cluster=cluster1 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ eval instance=0 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ instance=0 2024-09-16T23:34:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=test 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 1 10 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_1 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2024-09-16T23:34:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-16T23:34:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-16T23:34:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:34:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-16T23:34:12.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_1': 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 193fa3685952 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.193fa3685952 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:34:11 2024 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:34:11 2024 2024-09-16T23:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:34:11 2024 2024-09-16T23:34:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:34:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:34:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 3e7094e6-26a9-4ed7-91ec-14ec1a259e55 2024-09-16T23:34:12.136 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:34:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-16T23:34:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:34:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:34:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:34:12.140 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:34:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 4999168 bytes 2024-09-16T23:34:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 46892032 bytes 2024-09-16T23:34:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 129203200 bytes 2024-09-16T23:34:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 153726976 bytes 2024-09-16T23:34:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 200537088 bytes 2024-09-16T23:34:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 293147648 bytes 2024-09-16T23:34:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 352839680 bytes 2024-09-16T23:34:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 369210368 bytes 2024-09-16T23:34:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 387912704 bytes 2024-09-16T23:34:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 440375296 bytes 2024-09-16T23:34:13.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 492752896 bytes 2024-09-16T23:34:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 532718592 bytes 2024-09-16T23:34:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 607491072 bytes 2024-09-16T23:34:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 653814784 bytes 2024-09-16T23:34:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 667877376 bytes 2024-09-16T23:34:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 730109952 bytes 2024-09-16T23:34:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 730727424 bytes 2024-09-16T23:34:14.274 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 746107904 bytes 2024-09-16T23:34:14.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 775870464 bytes 2024-09-16T23:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 818512896 bytes 2024-09-16T23:34:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 843072512 bytes 2024-09-16T23:34:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 871873536 bytes 2024-09-16T23:34:14.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 987603968 bytes 2024-09-16T23:34:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 996093952 bytes 2024-09-16T23:34:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1067012096 bytes 2024-09-16T23:34:15.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1120764928 bytes 2024-09-16T23:34:15.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1143176192 bytes 2024-09-16T23:34:15.396 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1154504704 bytes 2024-09-16T23:34:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1268886528 bytes 2024-09-16T23:34:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1327001600 bytes 2024-09-16T23:34:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1355655168 bytes 2024-09-16T23:34:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1405688832 bytes 2024-09-16T23:34:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1426420736 bytes 2024-09-16T23:34:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1450496000 bytes 2024-09-16T23:34:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1493534720 bytes 2024-09-16T23:34:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1559407616 bytes 2024-09-16T23:34:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1595285504 bytes 2024-09-16T23:34:16.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1656483840 bytes 2024-09-16T23:34:16.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1724431360 bytes 2024-09-16T23:34:16.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1767293952 bytes 2024-09-16T23:34:16.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1805464576 bytes 2024-09-16T23:34:17.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1849401344 bytes 2024-09-16T23:34:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1889852416 bytes 2024-09-16T23:34:17.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1925630976 bytes 2024-09-16T23:34:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1947680768 bytes 2024-09-16T23:34:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1966961664 bytes 2024-09-16T23:34:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2036852736 bytes 2024-09-16T23:34:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2074683392 bytes 2024-09-16T23:34:17.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2140252160 bytes 2024-09-16T23:34:17.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2163621888 bytes 2024-09-16T23:34:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2259674112 bytes 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-16T23:34:18.145 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:18.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=14444b32075e 2024-09-16T23:34:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_1 present 14444b32075e 2024-09-16T23:34:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=14444b32075e 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=14444b32075e 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 14444b32075e 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 14444b32075e -o 14444b32075e = 14444b32075e 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-16T23:34:19.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:34:19.107+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556522ddc000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-16T23:34:19.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:34:19.107+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556522ddc000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-16T23:34:23.582 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:34:23.584+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6caa0f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:34:23.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:34:23.584+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0db900 [3/3e7094e6-26a9-4ed7-91ec-14ec1a259e55] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:34:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-16T23:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:24.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T23:34:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:34:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:34:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:34:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-16T23:34:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:34:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:24.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-16T23:34:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-16T23:34:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:34:24.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:34:24.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:34:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:34:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:34:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:34:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:34:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:34:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:34:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:34:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:34:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:34:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:34:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:34:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:31.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:34:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:34:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:34:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:34:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:34:39.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:34:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_1 2024-09-16T23:34:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_1 2024-09-16T23:34:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-16T23:34:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-16T23:34:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:34:56.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_2 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:34:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:34:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:34:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-16T23:34:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:34:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_2': 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1975e9597bd8 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1975e9597bd8 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:34:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:34:56 2024 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:34:56 2024 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:34:56 2024 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8cb7e36e-f272-412a-a197-a8d41aa6e283 2024-09-16T23:34:57.158 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:34:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-16T23:34:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:34:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:34:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:34:57.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:34:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:34:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:34:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:34:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 37225472 bytes 2024-09-16T23:34:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 97409024 bytes 2024-09-16T23:34:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 162492416 bytes 2024-09-16T23:34:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 220112896 bytes 2024-09-16T23:34:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 264209408 bytes 2024-09-16T23:34:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 290380800 bytes 2024-09-16T23:34:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 350596096 bytes 2024-09-16T23:34:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 437497856 bytes 2024-09-16T23:34:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 497240064 bytes 2024-09-16T23:34:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 540136448 bytes 2024-09-16T23:34:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 570146816 bytes 2024-09-16T23:34:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 610424832 bytes 2024-09-16T23:34:58.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 619474944 bytes 2024-09-16T23:34:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 620100608 bytes 2024-09-16T23:34:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 761176064 bytes 2024-09-16T23:34:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 766101504 bytes 2024-09-16T23:34:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 811730944 bytes 2024-09-16T23:34:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 831195136 bytes 2024-09-16T23:34:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 895854592 bytes 2024-09-16T23:34:59.538 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 915935232 bytes 2024-09-16T23:34:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 966526976 bytes 2024-09-16T23:34:59.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1035609088 bytes 2024-09-16T23:34:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1067969536 bytes 2024-09-16T23:34:59.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1068581888 bytes 2024-09-16T23:35:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1112067072 bytes 2024-09-16T23:35:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1167129600 bytes 2024-09-16T23:35:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1182202880 bytes 2024-09-16T23:35:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1223442432 bytes 2024-09-16T23:35:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1272255488 bytes 2024-09-16T23:35:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1310789632 bytes 2024-09-16T23:35:00.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1351824384 bytes 2024-09-16T23:35:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1367916544 bytes 2024-09-16T23:35:00.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1392813056 bytes 2024-09-16T23:35:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1449430016 bytes 2024-09-16T23:35:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1506940928 bytes 2024-09-16T23:35:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1586449408 bytes 2024-09-16T23:35:01.460 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1615491072 bytes 2024-09-16T23:35:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1630410752 bytes 2024-09-16T23:35:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1692398592 bytes 2024-09-16T23:35:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1727978496 bytes 2024-09-16T23:35:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1771954176 bytes 2024-09-16T23:35:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1772575744 bytes 2024-09-16T23:35:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1862759424 bytes 2024-09-16T23:35:02.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1909362688 bytes 2024-09-16T23:35:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1999513600 bytes 2024-09-16T23:35:02.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2071907328 bytes 2024-09-16T23:35:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2093756416 bytes 2024-09-16T23:35:02.941 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2124485632 bytes 2024-09-16T23:35:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2144465920 bytes 2024-09-16T23:35:03.135 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2165067776 bytes 2024-09-16T23:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:35:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:35:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:35:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=1444f2ff3b25 2024-09-16T23:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:35:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_2 present 1444f2ff3b25 2024-09-16T23:35:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=1444f2ff3b25 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:35:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=1444f2ff3b25 2024-09-16T23:35:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1444f2ff3b25 2024-09-16T23:35:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 1444f2ff3b25 -o 1444f2ff3b25 = 1444f2ff3b25 2024-09-16T23:35:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:35:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-16T23:35:04.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:35:04.132+0000 7f3e06797640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556524339c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-16T23:35:04.129 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:35:04.132+0000 7f3e06797640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556524339c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-16T23:35:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:35:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-16T23:35:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:09.142 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:35:09.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:09.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T23:35:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:35:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:35:09.188 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:35:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_2 2024-09-16T23:35:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_2 2024-09-16T23:35:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-16T23:35:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:09.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-16T23:35:09.332 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:09.333 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-16T23:35:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-16T23:35:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:35:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:09.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:09.426 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:35:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:35:11.418 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6508610506640294) > weight (0.25) 2024-09-16T23:35:11.418 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2548747824789722) > weight (0.25) 2024-09-16T23:35:11.418 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T23:35:11.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T23:35:11.420 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T23:35:11.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:35:11.423+0000 7f28c1fd1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 52940) UID: 1000 2024-09-16T23:35:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:12.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:35:14.420 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T23:35:14.421 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:35:14.421 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T23:35:14.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:35:14.421 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:35:14.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:35:14.425 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T23:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:16.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:16.797 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:35:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:24.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:24.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:35:30.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7751432964043468) > weight (0.25) 2024-09-16T23:35:30.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7208155758438706) > weight (0.25) 2024-09-16T23:35:30.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2683552037289011) > weight (0.25) 2024-09-16T23:35:30.594 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.2589478482557136) > weight (0.25) 2024-09-16T23:35:30.594 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-16T23:35:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_2 2024-09-16T23:35:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_2 2024-09-16T23:35:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-16T23:35:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-16T23:35:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:35:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_3 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:35:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:35:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:35:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:35:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2024-09-16T23:35:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-16T23:35:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:41.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:35:41.231 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-16T23:35:42.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:35:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_3': 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 19b5a46e66c2 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.19b5a46e66c2 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:35:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:35:41 2024 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:35:41 2024 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:35:41 2024 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8498c70a-1aeb-46ee-ba6c-cbd9efa0aaaa 2024-09-16T23:35:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:35:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-16T23:35:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:35:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=11 2024-09-16T23:35:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:35:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:35:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 45916160 bytes 2024-09-16T23:35:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 79102976 bytes 2024-09-16T23:35:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 131587072 bytes 2024-09-16T23:35:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 221863936 bytes 2024-09-16T23:35:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 291112960 bytes 2024-09-16T23:35:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 291695616 bytes 2024-09-16T23:35:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 334706688 bytes 2024-09-16T23:35:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 339485696 bytes 2024-09-16T23:35:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 430341120 bytes 2024-09-16T23:35:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 530191360 bytes 2024-09-16T23:35:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 602031104 bytes 2024-09-16T23:35:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 602620928 bytes 2024-09-16T23:35:43.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 690617344 bytes 2024-09-16T23:35:43.950 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 716971008 bytes 2024-09-16T23:35:44.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 777699328 bytes 2024-09-16T23:35:44.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 808765440 bytes 2024-09-16T23:35:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 828536832 bytes 2024-09-16T23:35:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 885505024 bytes 2024-09-16T23:35:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 913372160 bytes 2024-09-16T23:35:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 930255872 bytes 2024-09-16T23:35:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 937381888 bytes 2024-09-16T23:35:44.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1010445312 bytes 2024-09-16T23:35:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1057760256 bytes 2024-09-16T23:35:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1121414144 bytes 2024-09-16T23:35:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1185045504 bytes 2024-09-16T23:35:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1237018624 bytes 2024-09-16T23:35:45.430 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1271191552 bytes 2024-09-16T23:35:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1323370496 bytes 2024-09-16T23:35:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1331170304 bytes 2024-09-16T23:35:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1372487680 bytes 2024-09-16T23:35:45.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1415804928 bytes 2024-09-16T23:35:45.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1457940480 bytes 2024-09-16T23:35:46.128 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1531508736 bytes 2024-09-16T23:35:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1570736128 bytes 2024-09-16T23:35:46.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1596000256 bytes 2024-09-16T23:35:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1657413632 bytes 2024-09-16T23:35:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1689148416 bytes 2024-09-16T23:35:46.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1732124672 bytes 2024-09-16T23:35:46.789 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1759788032 bytes 2024-09-16T23:35:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1785289728 bytes 2024-09-16T23:35:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1834287104 bytes 2024-09-16T23:35:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1917709312 bytes 2024-09-16T23:35:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 2001683456 bytes 2024-09-16T23:35:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 2033129472 bytes 2024-09-16T23:35:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2095289344 bytes 2024-09-16T23:35:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2127327232 bytes 2024-09-16T23:35:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2145270784 bytes 2024-09-16T23:35:47.917 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2218064896 bytes 2024-09-16T23:35:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2246383616 bytes 2024-09-16T23:35:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2314580992 bytes 2024-09-16T23:35:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2320730112 bytes 2024-09-16T23:35:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2337116160 bytes 2024-09-16T23:35:48.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2357167104 bytes 2024-09-16T23:35:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2394723328 bytes 2024-09-16T23:35:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2438851584 bytes 2024-09-16T23:35:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2530869248 bytes 2024-09-16T23:35:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2561662976 bytes 2024-09-16T23:35:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2647394304 bytes 2024-09-16T23:35:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2659735552 bytes 2024-09-16T23:35:49.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2666725376 bytes 2024-09-16T23:35:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2745463808 bytes 2024-09-16T23:35:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2803586048 bytes 2024-09-16T23:35:49.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2868704256 bytes 2024-09-16T23:35:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2894579712 bytes 2024-09-16T23:35:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2928153600 bytes 2024-09-16T23:35:50.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 3011397632 bytes 2024-09-16T23:35:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 3064944640 bytes 2024-09-16T23:35:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3084473344 bytes 2024-09-16T23:35:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3151621120 bytes 2024-09-16T23:35:50.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3183332352 bytes 2024-09-16T23:35:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3183968256 bytes 2024-09-16T23:35:50.767 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3207580672 bytes 2024-09-16T23:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3225266176 bytes 2024-09-16T23:35:50.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3278533632 bytes 2024-09-16T23:35:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3313351680 bytes 2024-09-16T23:35:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3396682752 bytes 2024-09-16T23:35:51.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3466173440 bytes 2024-09-16T23:35:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3511305216 bytes 2024-09-16T23:35:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3561092096 bytes 2024-09-16T23:35:51.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3572731904 bytes 2024-09-16T23:35:51.843 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3590722560 bytes 2024-09-16T23:35:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3713052672 bytes 2024-09-16T23:35:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3776091136 bytes 2024-09-16T23:35:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3822618624 bytes 2024-09-16T23:35:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3861947392 bytes 2024-09-16T23:35:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3877434368 bytes 2024-09-16T23:35:52.629 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3914854400 bytes 2024-09-16T23:35:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3958032384 bytes 2024-09-16T23:35:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3976184832 bytes 2024-09-16T23:35:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 4036379648 bytes 2024-09-16T23:35:53.104 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 4067893248 bytes 2024-09-16T23:35:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 4105059328 bytes 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:35:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:35:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:35:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:35:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:35:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=1444a9b15867 2024-09-16T23:35:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:35:53.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:35:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_3 present 1444a9b15867 2024-09-16T23:35:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=1444a9b15867 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:35:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=1444a9b15867 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1444a9b15867 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 1444a9b15867 -o 1444a9b15867 = 1444a9b15867 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:35:53.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-16T23:35:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-16T23:35:53.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:35:53.767+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523b4a400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-16T23:35:53.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:35:53.767+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523b4a400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-16T23:35:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:35:58.779 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:58.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T23:35:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:35:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:35:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:35:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_3 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_3 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-16T23:35:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:35:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-16T23:35:58.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:35:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:35:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:35:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:35:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:35:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:35:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:35:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:35:59.106 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:35:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:35:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:35:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:36:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:36:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:36:00.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:36:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:36:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:36:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:36:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:36:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:36:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:36:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:36:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:36:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:36:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:36:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:36:14.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:36:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:36:14.435 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:36:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:36:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:14.540 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:36:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_3 2024-09-16T23:36:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_3 2024-09-16T23:36:30.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-16T23:36:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-16T23:36:30.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_4 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:36:30.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:36:30.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2024-09-16T23:36:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-16T23:36:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:36:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-16T23:36:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:36:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-16T23:36:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_4': 2024-09-16T23:36:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:36:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 19eb9c59720a 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.19eb9c59720a 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:36:30 2024 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:36:30 2024 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:36:30 2024 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:36:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:36:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 405e907f-1fc6-4d06-bd07-6e260fa4aac3 2024-09-16T23:36:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-16T23:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:36:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:36:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:36:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:36:31.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 46185472 bytes 2024-09-16T23:36:31.432 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 57779200 bytes 2024-09-16T23:36:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 93395968 bytes 2024-09-16T23:36:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 129162240 bytes 2024-09-16T23:36:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 129756160 bytes 2024-09-16T23:36:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 178562048 bytes 2024-09-16T23:36:31.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 204227584 bytes 2024-09-16T23:36:32.056 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 220537856 bytes 2024-09-16T23:36:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 265552896 bytes 2024-09-16T23:36:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 326978560 bytes 2024-09-16T23:36:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 355845120 bytes 2024-09-16T23:36:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 399991808 bytes 2024-09-16T23:36:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 481616896 bytes 2024-09-16T23:36:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 512748544 bytes 2024-09-16T23:36:32.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 552648704 bytes 2024-09-16T23:36:33.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 595793920 bytes 2024-09-16T23:36:33.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 626161664 bytes 2024-09-16T23:36:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 673946624 bytes 2024-09-16T23:36:33.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 717248512 bytes 2024-09-16T23:36:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 726000640 bytes 2024-09-16T23:36:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 735252480 bytes 2024-09-16T23:36:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 819474432 bytes 2024-09-16T23:36:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 871784448 bytes 2024-09-16T23:36:33.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 888148992 bytes 2024-09-16T23:36:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 983439360 bytes 2024-09-16T23:36:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1004213248 bytes 2024-09-16T23:36:34.279 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1017908224 bytes 2024-09-16T23:36:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1084590080 bytes 2024-09-16T23:36:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1096935424 bytes 2024-09-16T23:36:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1142781952 bytes 2024-09-16T23:36:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1214589952 bytes 2024-09-16T23:36:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1242928128 bytes 2024-09-16T23:36:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1251341312 bytes 2024-09-16T23:36:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1251949568 bytes 2024-09-16T23:36:35.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1283979264 bytes 2024-09-16T23:36:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1308972032 bytes 2024-09-16T23:36:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1354063872 bytes 2024-09-16T23:36:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1371726848 bytes 2024-09-16T23:36:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1457175552 bytes 2024-09-16T23:36:35.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1499855872 bytes 2024-09-16T23:36:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1552776192 bytes 2024-09-16T23:36:35.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1587650560 bytes 2024-09-16T23:36:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1657667584 bytes 2024-09-16T23:36:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1667529728 bytes 2024-09-16T23:36:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1668142080 bytes 2024-09-16T23:36:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1702776832 bytes 2024-09-16T23:36:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1757466624 bytes 2024-09-16T23:36:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1761902592 bytes 2024-09-16T23:36:36.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1810367488 bytes 2024-09-16T23:36:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1835711488 bytes 2024-09-16T23:36:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1911493632 bytes 2024-09-16T23:36:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 1941161984 bytes 2024-09-16T23:36:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 1965259776 bytes 2024-09-16T23:36:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 1965858816 bytes 2024-09-16T23:36:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 1987351552 bytes 2024-09-16T23:36:37.447 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2015090688 bytes 2024-09-16T23:36:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2054299648 bytes 2024-09-16T23:36:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2113855488 bytes 2024-09-16T23:36:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2160723968 bytes 2024-09-16T23:36:37.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2241231872 bytes 2024-09-16T23:36:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2252523520 bytes 2024-09-16T23:36:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2330480640 bytes 2024-09-16T23:36:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2398595072 bytes 2024-09-16T23:36:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2454716416 bytes 2024-09-16T23:36:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2488847360 bytes 2024-09-16T23:36:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2546410496 bytes 2024-09-16T23:36:38.737 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:36:38.739+0000 7fbcef505640 -1 reset not still connected to 0x56517cc9d790 2024-09-16T23:36:38.741 INFO:tasks.ceph.cluster2.osd.2.smithi098.stderr:2024-09-16T23:36:38.744+0000 7fbcef505640 -1 reset not still connected to 0x5651838a7ba0 2024-09-16T23:36:38.836 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2598296576 bytes 2024-09-16T23:36:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2677859328 bytes 2024-09-16T23:36:39.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2728258560 bytes 2024-09-16T23:36:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2769917952 bytes 2024-09-16T23:36:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2811973632 bytes 2024-09-16T23:36:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2911287296 bytes 2024-09-16T23:36:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2968124416 bytes 2024-09-16T23:36:39.824 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3030231040 bytes 2024-09-16T23:36:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3097636864 bytes 2024-09-16T23:36:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3098269696 bytes 2024-09-16T23:36:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3171665920 bytes 2024-09-16T23:36:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3193585664 bytes 2024-09-16T23:36:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3206544384 bytes 2024-09-16T23:36:40.425 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:36:40.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:36:40.426 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-16T23:36:40.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:36:40.430+0000 7f4dbfe79640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 59952) UID: 1000 2024-09-16T23:36:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3237607424 bytes 2024-09-16T23:36:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3265612800 bytes 2024-09-16T23:36:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3297913856 bytes 2024-09-16T23:36:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3344804864 bytes 2024-09-16T23:36:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3402740736 bytes 2024-09-16T23:36:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3453680640 bytes 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-16T23:36:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:36:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:36:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=1444b529bd1c 2024-09-16T23:36:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:36:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:36:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_4 present 1444b529bd1c 2024-09-16T23:36:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=1444b529bd1c 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:36:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:36:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=1444b529bd1c 2024-09-16T23:36:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1444b529bd1c 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 1444b529bd1c -o 1444b529bd1c = 1444b529bd1c 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:41.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-16T23:36:42.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:36:42.163+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523ef0c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-16T23:36:42.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:36:42.163+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523ef0c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-16T23:36:43.427 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:36:43.427 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:36:43.427 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:36:43.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:36:43.428 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:36:43.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:36:43.431 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-16T23:36:45.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:36:45.748+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca6d1860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:36:45.746 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:36:45.748+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca7f6a00 [3/405e907f-1fc6-4d06-bd07-6e260fa4aac3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:36:46.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8249588945204046) > weight (0.25) 2024-09-16T23:36:46.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5703761517322705) > weight (0.25) 2024-09-16T23:36:46.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5926264185565681) > weight (0.25) 2024-09-16T23:36:46.595 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.593482268478963) > weight (0.25) 2024-09-16T23:36:46.595 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-16T23:36:47.154 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:36:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:36:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:36:47.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T23:36:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:36:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:36:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-16T23:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:36:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_4 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_4 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-16T23:36:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:36:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-16T23:36:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:36:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:36:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:36:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:36:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-16T23:36:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-16T23:36:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-16T23:36:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:36:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:36:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:36:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:36:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:47.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:47.494 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:36:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:36:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:36:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:36:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:48.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:36:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:36:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:50.598 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:36:50.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:36:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:36:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:36:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:36:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:36:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:36:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:36:54.757 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:36:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:36:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:36:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:36:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:36:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:37:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:37:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:37:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:37:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:37:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:02.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:02.914 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:37:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_4 2024-09-16T23:37:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_4 2024-09-16T23:37:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-16T23:37:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-16T23:37:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_5 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:37:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:37:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-16T23:37:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:37:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-16T23:37:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_5': 2024-09-16T23:37:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:37:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:37:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:37:20.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1a2ab6fa9a9c 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1a2ab6fa9a9c 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:37:19 2024 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:37:19 2024 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:37:19 2024 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:37:20.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: f3a952d6-81a0-4682-824d-24a9829e53d0 2024-09-16T23:37:20.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:37:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-16T23:37:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:37:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=6 2024-09-16T23:37:20.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:37:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:37:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 22703104 bytes 2024-09-16T23:37:20.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 53227520 bytes 2024-09-16T23:37:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 53857280 bytes 2024-09-16T23:37:20.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 130185216 bytes 2024-09-16T23:37:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 154438656 bytes 2024-09-16T23:37:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 189759488 bytes 2024-09-16T23:37:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 238395392 bytes 2024-09-16T23:37:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 283509760 bytes 2024-09-16T23:37:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 311761920 bytes 2024-09-16T23:37:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 312375296 bytes 2024-09-16T23:37:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 389572608 bytes 2024-09-16T23:37:21.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 446654464 bytes 2024-09-16T23:37:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 481752064 bytes 2024-09-16T23:37:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 526802944 bytes 2024-09-16T23:37:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 578483200 bytes 2024-09-16T23:37:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 652476416 bytes 2024-09-16T23:37:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 766790656 bytes 2024-09-16T23:37:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 821572608 bytes 2024-09-16T23:37:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 868255744 bytes 2024-09-16T23:37:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 888388608 bytes 2024-09-16T23:37:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 941119488 bytes 2024-09-16T23:37:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1011960832 bytes 2024-09-16T23:37:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1083041792 bytes 2024-09-16T23:37:23.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1127985152 bytes 2024-09-16T23:37:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1184135168 bytes 2024-09-16T23:37:23.394 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1221347328 bytes 2024-09-16T23:37:23.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1261735936 bytes 2024-09-16T23:37:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1316936704 bytes 2024-09-16T23:37:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1378124800 bytes 2024-09-16T23:37:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1416164352 bytes 2024-09-16T23:37:24.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1483730944 bytes 2024-09-16T23:37:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1498787840 bytes 2024-09-16T23:37:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1619236864 bytes 2024-09-16T23:37:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1642901504 bytes 2024-09-16T23:37:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1658715136 bytes 2024-09-16T23:37:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1663754240 bytes 2024-09-16T23:37:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1688061952 bytes 2024-09-16T23:37:24.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1724646400 bytes 2024-09-16T23:37:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1748834304 bytes 2024-09-16T23:37:24.984 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1769189376 bytes 2024-09-16T23:37:25.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1823254528 bytes 2024-09-16T23:37:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1829138432 bytes 2024-09-16T23:37:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1927102464 bytes 2024-09-16T23:37:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1953228800 bytes 2024-09-16T23:37:25.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 2011480064 bytes 2024-09-16T23:37:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2064199680 bytes 2024-09-16T23:37:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2151548928 bytes 2024-09-16T23:37:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2188457984 bytes 2024-09-16T23:37:26.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2273346560 bytes 2024-09-16T23:37:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:37:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:37:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:37:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:37:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:37:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:37:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=1444ae65b216 2024-09-16T23:37:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:37:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_5 present 1444ae65b216 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=1444ae65b216 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:37:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:37:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:37:26.408 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:37:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=1444ae65b216 2024-09-16T23:37:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1444ae65b216 2024-09-16T23:37:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 1444ae65b216 -o 1444ae65b216 = 1444ae65b216 2024-09-16T23:37:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:37:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-16T23:37:26.801 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:37:26.803+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556524025c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-16T23:37:26.802 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:37:26.803+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556524025c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-16T23:37:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:37:31.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:37:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:37:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T23:37:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:37:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:37:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:37:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_5 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_5 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-16T23:37:31.960 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:37:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-16T23:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-16T23:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:37:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:37:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:37:32.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:37:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:37:33.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:37:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:33.346 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:37:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:37:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:37:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:37:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:37:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:37:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:35.545 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:37:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:37:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:37:39.603 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:39.687 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:37:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:37:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:37:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:37:47.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:37:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:37:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:38:00.431 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-16T23:38:00.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-16T23:38:00.433 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:38:00.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:38:00.437+0000 7f767d91f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 70334) UID: 1000 2024-09-16T23:38:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_5 2024-09-16T23:38:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_5 2024-09-16T23:38:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-16T23:38:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-16T23:38:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:38:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:38:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:38:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_6 2024-09-16T23:38:04.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2024-09-16T23:38:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-16T23:38:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:38:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-16T23:38:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:38:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-16T23:38:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_6': 2024-09-16T23:38:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:38:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:38:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:38:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1a608e880794 2024-09-16T23:38:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1a608e880794 2024-09-16T23:38:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:38:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:38:04 2024 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:38:04 2024 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:38:04 2024 2024-09-16T23:38:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:38:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:38:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 5687fc24-96eb-4ebd-9b28-3925a61d0bec 2024-09-16T23:38:05.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-16T23:38:05.202 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:38:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:38:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:38:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:38:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 62744576 bytes 2024-09-16T23:38:05.433 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-16T23:38:05.433 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:38:05.434 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-16T23:38:05.434 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-16T23:38:05.434 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-16T23:38:05.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-16T23:38:05.437 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-16T23:38:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 91725824 bytes 2024-09-16T23:38:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 115295232 bytes 2024-09-16T23:38:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 158577664 bytes 2024-09-16T23:38:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 171704320 bytes 2024-09-16T23:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 212924416 bytes 2024-09-16T23:38:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 294180864 bytes 2024-09-16T23:38:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 331613184 bytes 2024-09-16T23:38:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 358152192 bytes 2024-09-16T23:38:06.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 403371008 bytes 2024-09-16T23:38:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 470773760 bytes 2024-09-16T23:38:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 477536256 bytes 2024-09-16T23:38:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 518121472 bytes 2024-09-16T23:38:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 599365632 bytes 2024-09-16T23:38:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 623897600 bytes 2024-09-16T23:38:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 678946816 bytes 2024-09-16T23:38:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 694621184 bytes 2024-09-16T23:38:07.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 736586752 bytes 2024-09-16T23:38:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 759281664 bytes 2024-09-16T23:38:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 764322816 bytes 2024-09-16T23:38:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 846236672 bytes 2024-09-16T23:38:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 857355264 bytes 2024-09-16T23:38:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 896562176 bytes 2024-09-16T23:38:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 958645248 bytes 2024-09-16T23:38:08.168 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1002933248 bytes 2024-09-16T23:38:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1024063488 bytes 2024-09-16T23:38:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1047061504 bytes 2024-09-16T23:38:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1081051136 bytes 2024-09-16T23:38:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1081657344 bytes 2024-09-16T23:38:08.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1085771776 bytes 2024-09-16T23:38:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1109069824 bytes 2024-09-16T23:38:08.832 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1116883968 bytes 2024-09-16T23:38:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1139567616 bytes 2024-09-16T23:38:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1143938048 bytes 2024-09-16T23:38:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1177753600 bytes 2024-09-16T23:38:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1190289408 bytes 2024-09-16T23:38:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1214173184 bytes 2024-09-16T23:38:09.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1260616704 bytes 2024-09-16T23:38:09.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1346062336 bytes 2024-09-16T23:38:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1398444032 bytes 2024-09-16T23:38:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1421333504 bytes 2024-09-16T23:38:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1469544448 bytes 2024-09-16T23:38:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1522582528 bytes 2024-09-16T23:38:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1524146176 bytes 2024-09-16T23:38:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1536361472 bytes 2024-09-16T23:38:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1572489216 bytes 2024-09-16T23:38:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1575682048 bytes 2024-09-16T23:38:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1643270144 bytes 2024-09-16T23:38:10.837 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1763649536 bytes 2024-09-16T23:38:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1825276928 bytes 2024-09-16T23:38:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1871879168 bytes 2024-09-16T23:38:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 1893555200 bytes 2024-09-16T23:38:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 1907614720 bytes 2024-09-16T23:38:11.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 1929611264 bytes 2024-09-16T23:38:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 1980084224 bytes 2024-09-16T23:38:11.635 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2030055424 bytes 2024-09-16T23:38:11.726 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2062167040 bytes 2024-09-16T23:38:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2113399808 bytes 2024-09-16T23:38:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2187210752 bytes 2024-09-16T23:38:12.117 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2215058432 bytes 2024-09-16T23:38:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2219000832 bytes 2024-09-16T23:38:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2264171520 bytes 2024-09-16T23:38:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2264799232 bytes 2024-09-16T23:38:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2293277696 bytes 2024-09-16T23:38:12.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2317850624 bytes 2024-09-16T23:38:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2358301696 bytes 2024-09-16T23:38:12.809 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2358931456 bytes 2024-09-16T23:38:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2359535616 bytes 2024-09-16T23:38:12.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2396740608 bytes 2024-09-16T23:38:13.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2439569408 bytes 2024-09-16T23:38:13.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2478577664 bytes 2024-09-16T23:38:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2479185920 bytes 2024-09-16T23:38:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2568930304 bytes 2024-09-16T23:38:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2597673984 bytes 2024-09-16T23:38:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2619142144 bytes 2024-09-16T23:38:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2691602432 bytes 2024-09-16T23:38:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 2714934272 bytes 2024-09-16T23:38:14.019 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 2757872640 bytes 2024-09-16T23:38:14.134 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 2775162880 bytes 2024-09-16T23:38:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 2817727488 bytes 2024-09-16T23:38:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 2890371072 bytes 2024-09-16T23:38:14.504 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 2921516032 bytes 2024-09-16T23:38:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 2922131456 bytes 2024-09-16T23:38:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3001867264 bytes 2024-09-16T23:38:14.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3026475008 bytes 2024-09-16T23:38:14.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3027120128 bytes 2024-09-16T23:38:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3085703168 bytes 2024-09-16T23:38:15.141 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3093436416 bytes 2024-09-16T23:38:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3163299840 bytes 2024-09-16T23:38:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3191945216 bytes 2024-09-16T23:38:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3200794624 bytes 2024-09-16T23:38:15.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3264264192 bytes 2024-09-16T23:38:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3289857024 bytes 2024-09-16T23:38:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3321372672 bytes 2024-09-16T23:38:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3400529920 bytes 2024-09-16T23:38:16.092 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3454473216 bytes 2024-09-16T23:38:16.229 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 3518411776 bytes 2024-09-16T23:38:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 3574657024 bytes 2024-09-16T23:38:16.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 3634918400 bytes 2024-09-16T23:38:16.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 3661776896 bytes 2024-09-16T23:38:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 3717426176 bytes 2024-09-16T23:38:16.844 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 3770780672 bytes 2024-09-16T23:38:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 3825950720 bytes 2024-09-16T23:38:17.089 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 3876957184 bytes 2024-09-16T23:38:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 3926026240 bytes 2024-09-16T23:38:17.309 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 3941026816 bytes 2024-09-16T23:38:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 3978950656 bytes 2024-09-16T23:38:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4010585088 bytes 2024-09-16T23:38:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4035549184 bytes 2024-09-16T23:38:17.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11000 IOs, 4040521728 bytes 2024-09-16T23:38:17.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11100 IOs, 4135475200 bytes 2024-09-16T23:38:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11200 IOs, 4218090496 bytes 2024-09-16T23:38:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:11300 IOs, 4244864000 bytes 2024-09-16T23:38:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:38:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:38:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:38:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:38:18.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:38:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:38:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=144429afe190 2024-09-16T23:38:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:38:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:38:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_6 present 144429afe190 2024-09-16T23:38:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=144429afe190 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:38:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:38:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:38:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=144429afe190 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 144429afe190 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 144429afe190 -o 144429afe190 = 144429afe190 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-16T23:38:19.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:38:19.169+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5565237a3c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-16T23:38:19.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:38:19.169+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5565237a3c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-16T23:38:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:38:23.585+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:38:23.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:38:23.585+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca0db900 [3/5687fc24-96eb-4ebd-9b28-3925a61d0bec] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:38:24.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:38:24.179 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:38:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:38:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:38:24.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:38:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T23:38:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:38:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:38:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_6 2024-09-16T23:38:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:38:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:38:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_6 2024-09-16T23:38:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-16T23:38:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:38:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-16T23:38:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:38:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:38:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:38:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:38:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-16T23:38:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:38:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:38:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:38:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:38:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:38:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:25.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:38:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:38:25.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:38:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:38:26.596 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-16T23:38:26.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-16T23:38:26.597 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:38:26.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:38:26.600+0000 7f58cfb43640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47166) UID: 1000 2024-09-16T23:38:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:38:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:38:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:38:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:38:31.598 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-16T23:38:31.598 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:38:31.598 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-16T23:38:31.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-16T23:38:31.599 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-16T23:38:31.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-16T23:38:31.602 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-16T23:38:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:38:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:38:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:38:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:38:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:38:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:38:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:38:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:38:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_6 2024-09-16T23:38:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_6 2024-09-16T23:38:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-16T23:38:55.979 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-16T23:38:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_7 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:38:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:38:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-16T23:38:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:38:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_7': 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1aa3ccf80353 2024-09-16T23:38:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1aa3ccf80353 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:38:56 2024 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:38:56 2024 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:38:56 2024 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8c5451de-1447-4ec8-b5e6-6f1342c7db1b 2024-09-16T23:38:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:38:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-16T23:38:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:38:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:38:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:38:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:38:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=13 2024-09-16T23:38:57.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:38:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:38:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 53895168 bytes 2024-09-16T23:38:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 88892416 bytes 2024-09-16T23:38:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 118910976 bytes 2024-09-16T23:38:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 132579328 bytes 2024-09-16T23:38:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 189809664 bytes 2024-09-16T23:38:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 247724032 bytes 2024-09-16T23:38:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 262562816 bytes 2024-09-16T23:38:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 291522560 bytes 2024-09-16T23:38:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 350917632 bytes 2024-09-16T23:38:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 370716672 bytes 2024-09-16T23:38:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 412717056 bytes 2024-09-16T23:38:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 528644096 bytes 2024-09-16T23:38:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 550275072 bytes 2024-09-16T23:38:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 600768512 bytes 2024-09-16T23:38:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 653851648 bytes 2024-09-16T23:38:59.183 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 729865216 bytes 2024-09-16T23:38:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 789101568 bytes 2024-09-16T23:38:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 804277248 bytes 2024-09-16T23:38:59.549 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 874748928 bytes 2024-09-16T23:38:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 896773120 bytes 2024-09-16T23:38:59.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 921076736 bytes 2024-09-16T23:38:59.899 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 984914944 bytes 2024-09-16T23:38:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1004148736 bytes 2024-09-16T23:39:00.132 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1069490176 bytes 2024-09-16T23:39:00.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1083701248 bytes 2024-09-16T23:39:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1111375872 bytes 2024-09-16T23:39:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1145126912 bytes 2024-09-16T23:39:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1161937920 bytes 2024-09-16T23:39:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1197635584 bytes 2024-09-16T23:39:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1222857728 bytes 2024-09-16T23:39:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1263063040 bytes 2024-09-16T23:39:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1297915904 bytes 2024-09-16T23:39:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1334159360 bytes 2024-09-16T23:39:01.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1360848896 bytes 2024-09-16T23:39:01.502 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1447016448 bytes 2024-09-16T23:39:01.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1505062912 bytes 2024-09-16T23:39:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1505696768 bytes 2024-09-16T23:39:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1546317824 bytes 2024-09-16T23:39:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1590706176 bytes 2024-09-16T23:39:02.098 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1662790656 bytes 2024-09-16T23:39:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1676574720 bytes 2024-09-16T23:39:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1711851520 bytes 2024-09-16T23:39:02.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1723285504 bytes 2024-09-16T23:39:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1746321408 bytes 2024-09-16T23:39:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1746946048 bytes 2024-09-16T23:39:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1747545088 bytes 2024-09-16T23:39:02.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1784625152 bytes 2024-09-16T23:39:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1822025728 bytes 2024-09-16T23:39:02.996 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1849944064 bytes 2024-09-16T23:39:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1870944256 bytes 2024-09-16T23:39:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1890501632 bytes 2024-09-16T23:39:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 1908774912 bytes 2024-09-16T23:39:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 1923082240 bytes 2024-09-16T23:39:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 1960585216 bytes 2024-09-16T23:39:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 1989334016 bytes 2024-09-16T23:39:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2031369216 bytes 2024-09-16T23:39:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2069562368 bytes 2024-09-16T23:39:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2110862336 bytes 2024-09-16T23:39:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2150494208 bytes 2024-09-16T23:39:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2176985088 bytes 2024-09-16T23:39:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2186421248 bytes 2024-09-16T23:39:04.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2238979072 bytes 2024-09-16T23:39:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2307933184 bytes 2024-09-16T23:39:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2337579008 bytes 2024-09-16T23:39:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2399507456 bytes 2024-09-16T23:39:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2456415232 bytes 2024-09-16T23:39:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2502884352 bytes 2024-09-16T23:39:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2535986176 bytes 2024-09-16T23:39:05.239 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2576878592 bytes 2024-09-16T23:39:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2610475008 bytes 2024-09-16T23:39:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2655724544 bytes 2024-09-16T23:39:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2694122496 bytes 2024-09-16T23:39:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2756402176 bytes 2024-09-16T23:39:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2757025792 bytes 2024-09-16T23:39:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2843435008 bytes 2024-09-16T23:39:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2868110336 bytes 2024-09-16T23:39:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 2888910848 bytes 2024-09-16T23:39:06.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 2930051072 bytes 2024-09-16T23:39:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 2948407296 bytes 2024-09-16T23:39:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3009956864 bytes 2024-09-16T23:39:06.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3054420992 bytes 2024-09-16T23:39:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3124197376 bytes 2024-09-16T23:39:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3185212416 bytes 2024-09-16T23:39:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3272353792 bytes 2024-09-16T23:39:07.178 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3334132736 bytes 2024-09-16T23:39:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3394099200 bytes 2024-09-16T23:39:07.429 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3439574016 bytes 2024-09-16T23:39:07.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3446521856 bytes 2024-09-16T23:39:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3464248320 bytes 2024-09-16T23:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3551770624 bytes 2024-09-16T23:39:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3648826368 bytes 2024-09-16T23:39:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3674128384 bytes 2024-09-16T23:39:08.175 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3751476224 bytes 2024-09-16T23:39:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3766433792 bytes 2024-09-16T23:39:08.384 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 3789151232 bytes 2024-09-16T23:39:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 3909986304 bytes 2024-09-16T23:39:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4024793088 bytes 2024-09-16T23:39:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4108381184 bytes 2024-09-16T23:39:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4112244736 bytes 2024-09-16T23:39:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4155415552 bytes 2024-09-16T23:39:09.209 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10100 IOs, 4178841600 bytes 2024-09-16T23:39:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10200 IOs, 4200627200 bytes 2024-09-16T23:39:09.406 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10300 IOs, 4246878208 bytes 2024-09-16T23:39:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10400 IOs, 4304520192 bytes 2024-09-16T23:39:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10500 IOs, 4322413568 bytes 2024-09-16T23:39:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10600 IOs, 4333543424 bytes 2024-09-16T23:39:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10700 IOs, 4382014464 bytes 2024-09-16T23:39:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10800 IOs, 4440290304 bytes 2024-09-16T23:39:10.148 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10900 IOs, 4523085824 bytes 2024-09-16T23:39:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:39:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:39:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:39:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:39:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-16T23:39:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:39:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:39:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:39:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:39:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:39:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:39:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:39:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:39:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=144484a5fae5 2024-09-16T23:39:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:39:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:39:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_7 present 144484a5fae5 2024-09-16T23:39:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:39:10.412 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=144484a5fae5 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:39:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:39:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:39:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:39:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:39:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=144484a5fae5 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 144484a5fae5 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 144484a5fae5 -o 144484a5fae5 = 144484a5fae5 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-16T23:39:10.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:39:10.844+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523777400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-16T23:39:10.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:39:10.844+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556523777400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-16T23:39:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:39:15.753+0000 7f6dc3f71640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca900000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:39:15.751 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:39:15.753+0000 7f6dcaf7f640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810780 [3/8c5451de-1447-4ec8-b5e6-6f1342c7db1b] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:39:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:39:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:39:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:39:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:39:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:39:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:39:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T23:39:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:39:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:39:15.887 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-16T23:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:15.989 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_7 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_7 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-16T23:39:15.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:39:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-16T23:39:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:39:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-16T23:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-16T23:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:39:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:39:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:39:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:39:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:17.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:39:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:39:17.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:39:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:17.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:39:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:19.325 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:39:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:39:19.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:39:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:19.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:39:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:23.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:23.431 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:39:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:39:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:39:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:23.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:39:31.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8743476641015274) > weight (0.25) 2024-09-16T23:39:31.439 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.47415459423082584) > weight (0.25) 2024-09-16T23:39:31.439 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-16T23:39:31.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-16T23:39:31.440 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-16T23:39:31.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:39:31.443+0000 7f8b0d96d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 69216) UID: 1000 2024-09-16T23:39:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:39:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:39:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:39:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:39:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:39:36.443 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-16T23:39:36.444 INFO:teuthology.orchestra.run:waiting for 600 2024-09-16T23:39:36.444 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-16T23:39:36.444 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-16T23:39:36.444 DEBUG:teuthology.orchestra.run.smithi098:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-16T23:39:36.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-16T23:39:36.448 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2024-09-16T23:39:40.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.42939912229355326) > weight (0.25) 2024-09-16T23:39:40.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.26666751973951885) > weight (0.25) 2024-09-16T23:39:40.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8472686995457874) > weight (0.25) 2024-09-16T23:39:40.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.747712526315952) > weight (0.25) 2024-09-16T23:39:40.603 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-16T23:39:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_7 2024-09-16T23:39:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_7 2024-09-16T23:39:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-16T23:39:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-16T23:39:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:39:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:39:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:39:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_8 2024-09-16T23:39:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:39:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:39:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-16T23:39:48.149 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:39:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-16T23:39:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_8': 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1ae22b0f9fe0 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1ae22b0f9fe0 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:39:47 2024 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:39:47 2024 2024-09-16T23:39:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:39:47 2024 2024-09-16T23:39:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:39:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:39:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 8456952e-8c9c-4a2c-baaa-04a8f08dc77f 2024-09-16T23:39:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:39:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-16T23:39:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:39:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:39:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:39:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:39:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=12 2024-09-16T23:39:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:39:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:39:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 26963968 bytes 2024-09-16T23:39:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 41746432 bytes 2024-09-16T23:39:48.590 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 146489344 bytes 2024-09-16T23:39:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 192444416 bytes 2024-09-16T23:39:48.805 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 213965824 bytes 2024-09-16T23:39:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 241099776 bytes 2024-09-16T23:39:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 255894528 bytes 2024-09-16T23:39:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 297310208 bytes 2024-09-16T23:39:49.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 317428736 bytes 2024-09-16T23:39:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 441730048 bytes 2024-09-16T23:39:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 487340032 bytes 2024-09-16T23:39:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 528372736 bytes 2024-09-16T23:39:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 570038272 bytes 2024-09-16T23:39:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 606790656 bytes 2024-09-16T23:39:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 630486016 bytes 2024-09-16T23:39:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 695235584 bytes 2024-09-16T23:39:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 702577664 bytes 2024-09-16T23:39:50.535 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 807041024 bytes 2024-09-16T23:39:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 851717120 bytes 2024-09-16T23:39:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 878859264 bytes 2024-09-16T23:39:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 990913536 bytes 2024-09-16T23:39:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 1050641408 bytes 2024-09-16T23:39:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1091396608 bytes 2024-09-16T23:39:51.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1165485056 bytes 2024-09-16T23:39:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1213860864 bytes 2024-09-16T23:39:51.585 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1257180160 bytes 2024-09-16T23:39:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1261164544 bytes 2024-09-16T23:39:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1311284224 bytes 2024-09-16T23:39:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1371638784 bytes 2024-09-16T23:39:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1405099008 bytes 2024-09-16T23:39:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1469780992 bytes 2024-09-16T23:39:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1508672512 bytes 2024-09-16T23:39:52.382 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1527944192 bytes 2024-09-16T23:39:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1549602816 bytes 2024-09-16T23:39:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1611701248 bytes 2024-09-16T23:39:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1655609344 bytes 2024-09-16T23:39:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1705684992 bytes 2024-09-16T23:39:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1798431744 bytes 2024-09-16T23:39:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1817391104 bytes 2024-09-16T23:39:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1844583424 bytes 2024-09-16T23:39:53.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1853086720 bytes 2024-09-16T23:39:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1886724096 bytes 2024-09-16T23:39:53.514 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1901614080 bytes 2024-09-16T23:39:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1939869696 bytes 2024-09-16T23:39:53.756 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1995368448 bytes 2024-09-16T23:39:53.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 2030004224 bytes 2024-09-16T23:39:53.988 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 2067324928 bytes 2024-09-16T23:39:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2190513152 bytes 2024-09-16T23:39:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2207685632 bytes 2024-09-16T23:39:54.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2241629184 bytes 2024-09-16T23:39:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2243580928 bytes 2024-09-16T23:39:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2357996544 bytes 2024-09-16T23:39:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2418464768 bytes 2024-09-16T23:39:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2460223488 bytes 2024-09-16T23:39:55.016 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2509511680 bytes 2024-09-16T23:39:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2565953536 bytes 2024-09-16T23:39:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2605917184 bytes 2024-09-16T23:39:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2633106432 bytes 2024-09-16T23:39:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2693587968 bytes 2024-09-16T23:39:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2709904384 bytes 2024-09-16T23:39:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2715673600 bytes 2024-09-16T23:39:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2752728064 bytes 2024-09-16T23:39:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2803811328 bytes 2024-09-16T23:39:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2808128512 bytes 2024-09-16T23:39:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2877189120 bytes 2024-09-16T23:39:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2896254976 bytes 2024-09-16T23:39:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2917837824 bytes 2024-09-16T23:39:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2960604160 bytes 2024-09-16T23:39:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3009845248 bytes 2024-09-16T23:39:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3122056192 bytes 2024-09-16T23:39:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3152513024 bytes 2024-09-16T23:39:57.000 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3214536704 bytes 2024-09-16T23:39:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3221979136 bytes 2024-09-16T23:39:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3330206720 bytes 2024-09-16T23:39:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3345775616 bytes 2024-09-16T23:39:57.445 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3346399232 bytes 2024-09-16T23:39:57.548 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3372278784 bytes 2024-09-16T23:39:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3443784704 bytes 2024-09-16T23:39:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3458201600 bytes 2024-09-16T23:39:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3485588480 bytes 2024-09-16T23:39:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3529080832 bytes 2024-09-16T23:39:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3529698304 bytes 2024-09-16T23:39:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3593609216 bytes 2024-09-16T23:39:58.323 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3594193920 bytes 2024-09-16T23:39:58.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3594812416 bytes 2024-09-16T23:39:58.541 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3649611776 bytes 2024-09-16T23:39:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3697733632 bytes 2024-09-16T23:39:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3714488320 bytes 2024-09-16T23:39:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3744070656 bytes 2024-09-16T23:39:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3845964800 bytes 2024-09-16T23:39:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3878257664 bytes 2024-09-16T23:39:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9200 IOs, 3901554688 bytes 2024-09-16T23:39:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9300 IOs, 3964619776 bytes 2024-09-16T23:39:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9400 IOs, 3995750400 bytes 2024-09-16T23:39:59.619 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9500 IOs, 4054067200 bytes 2024-09-16T23:39:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9600 IOs, 4078153728 bytes 2024-09-16T23:39:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9700 IOs, 4093160448 bytes 2024-09-16T23:39:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9800 IOs, 4137377792 bytes 2024-09-16T23:40:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9900 IOs, 4143187968 bytes 2024-09-16T23:40:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:10000 IOs, 4153267200 bytes 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:00.204 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:40:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:40:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=1444bb371af0 2024-09-16T23:40:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:40:00.249 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:40:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_8 present 1444bb371af0 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=1444bb371af0 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:40:00.351 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:40:00.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=1444bb371af0 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1444bb371af0 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 1444bb371af0 -o 1444bb371af0 = 1444bb371af0 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:40:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-16T23:40:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-16T23:40:01.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:40:01.176+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5565235acc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-16T23:40:01.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:40:01.176+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5565235acc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-16T23:40:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:40:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:40:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:40:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T23:40:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:40:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:40:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_8 2024-09-16T23:40:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_8 2024-09-16T23:40:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-16T23:40:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:06.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-16T23:40:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:06.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:40:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:06.462 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:40:09.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:09.713 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:40:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:13.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:40:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:40:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_8 2024-09-16T23:40:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_8 2024-09-16T23:40:37.924 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-16T23:40:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-16T23:40:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:40:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_9 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:40:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:40:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-16T23:40:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:40:39.227 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_9': 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1b189c3c5de0 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1b189c3c5de0 2024-09-16T23:40:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:40:38 2024 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:40:38 2024 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:40:38 2024 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: 785ee54e-d1c3-49bf-966e-24cd8ea006d0 2024-09-16T23:40:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:40:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-16T23:40:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:40:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=10 2024-09-16T23:40:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:40:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:40:39.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 590848 bytes 2024-09-16T23:40:39.515 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 48546816 bytes 2024-09-16T23:40:39.600 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 49176576 bytes 2024-09-16T23:40:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 104606720 bytes 2024-09-16T23:40:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 189227008 bytes 2024-09-16T23:40:39.990 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 246356992 bytes 2024-09-16T23:40:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 296253440 bytes 2024-09-16T23:40:40.233 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 343593984 bytes 2024-09-16T23:40:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 359410688 bytes 2024-09-16T23:40:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 424888320 bytes 2024-09-16T23:40:40.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 446771200 bytes 2024-09-16T23:40:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 496690176 bytes 2024-09-16T23:40:40.802 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 550966272 bytes 2024-09-16T23:40:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 599074816 bytes 2024-09-16T23:40:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 634117120 bytes 2024-09-16T23:40:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 685670400 bytes 2024-09-16T23:40:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 750169088 bytes 2024-09-16T23:40:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 809312256 bytes 2024-09-16T23:40:41.524 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 813497344 bytes 2024-09-16T23:40:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 840783872 bytes 2024-09-16T23:40:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 912472064 bytes 2024-09-16T23:40:41.912 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 975116288 bytes 2024-09-16T23:40:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 1020922880 bytes 2024-09-16T23:40:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 1133132800 bytes 2024-09-16T23:40:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 1142694912 bytes 2024-09-16T23:40:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 1165870080 bytes 2024-09-16T23:40:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 1166504960 bytes 2024-09-16T23:40:42.561 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 1167757312 bytes 2024-09-16T23:40:42.674 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1204122624 bytes 2024-09-16T23:40:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1250452480 bytes 2024-09-16T23:40:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1256223744 bytes 2024-09-16T23:40:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1315385344 bytes 2024-09-16T23:40:43.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1345315840 bytes 2024-09-16T23:40:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1411973120 bytes 2024-09-16T23:40:43.365 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1454905344 bytes 2024-09-16T23:40:43.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1519633408 bytes 2024-09-16T23:40:43.641 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1582287872 bytes 2024-09-16T23:40:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1603747840 bytes 2024-09-16T23:40:43.868 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1651425280 bytes 2024-09-16T23:40:43.983 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1694202880 bytes 2024-09-16T23:40:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1743039488 bytes 2024-09-16T23:40:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1800146944 bytes 2024-09-16T23:40:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1840677888 bytes 2024-09-16T23:40:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1856675840 bytes 2024-09-16T23:40:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1878199296 bytes 2024-09-16T23:40:44.679 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1916742656 bytes 2024-09-16T23:40:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1944419328 bytes 2024-09-16T23:40:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 2054914048 bytes 2024-09-16T23:40:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 2141852672 bytes 2024-09-16T23:40:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 2185373696 bytes 2024-09-16T23:40:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 2220002304 bytes 2024-09-16T23:40:45.434 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2237236224 bytes 2024-09-16T23:40:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2273402880 bytes 2024-09-16T23:40:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2396264448 bytes 2024-09-16T23:40:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2400432128 bytes 2024-09-16T23:40:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2436109312 bytes 2024-09-16T23:40:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2494586880 bytes 2024-09-16T23:40:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2544278528 bytes 2024-09-16T23:40:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2580399104 bytes 2024-09-16T23:40:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2596867072 bytes 2024-09-16T23:40:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2696011776 bytes 2024-09-16T23:40:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2701622272 bytes 2024-09-16T23:40:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2735027200 bytes 2024-09-16T23:40:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2800872448 bytes 2024-09-16T23:40:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2850507776 bytes 2024-09-16T23:40:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2926666752 bytes 2024-09-16T23:40:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2993871872 bytes 2024-09-16T23:40:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 3038700544 bytes 2024-09-16T23:40:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 3071417344 bytes 2024-09-16T23:40:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 3152208896 bytes 2024-09-16T23:40:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 3191624704 bytes 2024-09-16T23:40:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 3239144448 bytes 2024-09-16T23:40:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 3276670976 bytes 2024-09-16T23:40:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 3346889728 bytes 2024-09-16T23:40:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 3374763008 bytes 2024-09-16T23:40:48.366 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 3387958272 bytes 2024-09-16T23:40:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3461544960 bytes 2024-09-16T23:40:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3524670464 bytes 2024-09-16T23:40:48.770 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3574140928 bytes 2024-09-16T23:40:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3575377920 bytes 2024-09-16T23:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3625529344 bytes 2024-09-16T23:40:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3640281088 bytes 2024-09-16T23:40:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3679815680 bytes 2024-09-16T23:40:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3685799936 bytes 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:40:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:40:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=144490391106 2024-09-16T23:40:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:40:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:40:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_9 present 144490391106 2024-09-16T23:40:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=144490391106 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:40:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=144490391106 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 144490391106 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 144490391106 -o 144490391106 = 144490391106 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-16T23:40:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-16T23:40:49.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:40:49.848+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556521c0d000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-16T23:40:49.846 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:40:49.848+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556521c0d000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-16T23:40:53.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:40:53.586+0000 7f6dc4772640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c6ca901a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-16T23:40:53.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:40:53.586+0000 7f6dca77e640 -1 rbd::mirror::ImageReplayer: 0x55c6ca810780 [3/785ee54e-d1c3-49bf-966e-24cd8ea006d0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-16T23:40:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:40:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:40:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:40:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T23:40:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:40:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:40:54.904 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:40:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_9 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_9 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-16T23:40:55.007 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:40:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-16T23:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-16T23:40:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-16T23:40:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-16T23:40:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:40:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:40:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:40:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:40:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:40:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:40:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:40:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:40:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:40:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:40:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:40:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:40:58.355 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:40:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:40:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:40:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:41:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:41:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:41:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:41:02.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:41:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:41:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:41:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:41:10.716 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:41:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:41:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:41:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_9 2024-09-16T23:41:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_9 2024-09-16T23:41:27.028 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-16T23:41:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-16T23:41:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for i in `seq 1 10` 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image=image_10 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local size=128 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -n 512M ']' 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ size=512M 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ shift 2024-09-16T23:41:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_10 2024-09-16T23:41:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-16T23:41:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local mode=snapshot 2024-09-16T23:41:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-16T23:41:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring enabled 2024-09-16T23:41:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:rbd image 'image_10': 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: size 512 MiB in 128 objects 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: order 22 (4 MiB objects) 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: snapshot_count: 1 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: id: 1b4e1bde97bd 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: block_name_prefix: rbd_data.1b4e1bde97bd 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: format: 2 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: features: layering 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: op_features: 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: flags: 2024-09-16T23:41:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: create_timestamp: Mon Sep 16 23:41:27 2024 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: access_timestamp: Mon Sep 16 23:41:27 2024 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: modify_timestamp: Mon Sep 16 23:41:27 2024 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring state: enabled 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring mode: snapshot 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring global id: b84c5831-0b5e-4763-882a-eb398e0e5abe 2024-09-16T23:41:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout: mirroring primary: true 2024-09-16T23:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-16T23:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-16T23:41:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local duration=11 2024-09-16T23:41:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:41:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-16T23:41:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:100 IOs, 56445952 bytes 2024-09-16T23:41:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:200 IOs, 122967040 bytes 2024-09-16T23:41:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:300 IOs, 144423936 bytes 2024-09-16T23:41:28.703 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:400 IOs, 145062912 bytes 2024-09-16T23:41:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:500 IOs, 203206656 bytes 2024-09-16T23:41:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:600 IOs, 244922368 bytes 2024-09-16T23:41:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:700 IOs, 257632256 bytes 2024-09-16T23:41:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:800 IOs, 273892352 bytes 2024-09-16T23:41:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:900 IOs, 282434560 bytes 2024-09-16T23:41:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1000 IOs, 316792832 bytes 2024-09-16T23:41:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1100 IOs, 342432768 bytes 2024-09-16T23:41:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1200 IOs, 367652864 bytes 2024-09-16T23:41:29.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3565524954916661) > weight (0.25) 2024-09-16T23:41:29.604 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5058394983788971) > weight (0.25) 2024-09-16T23:41:29.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7922363580742413) > weight (0.25) 2024-09-16T23:41:29.605 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8477996283339216) > weight (0.25) 2024-09-16T23:41:29.605 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2024-09-16T23:41:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1300 IOs, 440438784 bytes 2024-09-16T23:41:29.786 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1400 IOs, 506442752 bytes 2024-09-16T23:41:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1500 IOs, 552119296 bytes 2024-09-16T23:41:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1600 IOs, 606958592 bytes 2024-09-16T23:41:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1700 IOs, 665238528 bytes 2024-09-16T23:41:30.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1800 IOs, 687537152 bytes 2024-09-16T23:41:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:1900 IOs, 717645824 bytes 2024-09-16T23:41:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2000 IOs, 734860288 bytes 2024-09-16T23:41:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2100 IOs, 767568896 bytes 2024-09-16T23:41:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2200 IOs, 782154752 bytes 2024-09-16T23:41:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2300 IOs, 832179200 bytes 2024-09-16T23:41:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2400 IOs, 885321728 bytes 2024-09-16T23:41:31.011 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2500 IOs, 900055040 bytes 2024-09-16T23:41:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2600 IOs, 924352512 bytes 2024-09-16T23:41:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2700 IOs, 956639232 bytes 2024-09-16T23:41:31.499 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2800 IOs, 970972160 bytes 2024-09-16T23:41:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:2900 IOs, 1027240960 bytes 2024-09-16T23:41:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3000 IOs, 1058985984 bytes 2024-09-16T23:41:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3100 IOs, 1150970880 bytes 2024-09-16T23:41:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3200 IOs, 1208908800 bytes 2024-09-16T23:41:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3300 IOs, 1257527296 bytes 2024-09-16T23:41:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3400 IOs, 1295327232 bytes 2024-09-16T23:41:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3500 IOs, 1308596224 bytes 2024-09-16T23:41:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3600 IOs, 1333781504 bytes 2024-09-16T23:41:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3700 IOs, 1361750016 bytes 2024-09-16T23:41:32.717 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3800 IOs, 1420670976 bytes 2024-09-16T23:41:32.799 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:3900 IOs, 1427729408 bytes 2024-09-16T23:41:32.942 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4000 IOs, 1494656000 bytes 2024-09-16T23:41:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4100 IOs, 1554960384 bytes 2024-09-16T23:41:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4200 IOs, 1580678144 bytes 2024-09-16T23:41:33.301 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4300 IOs, 1634395136 bytes 2024-09-16T23:41:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4400 IOs, 1719928832 bytes 2024-09-16T23:41:33.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9298736912755061) > weight (0.25) 2024-09-16T23:41:33.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.466864512888048) > weight (0.25) 2024-09-16T23:41:33.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5220599926271802) > weight (0.25) 2024-09-16T23:41:33.449 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5622736180728963) > weight (0.25) 2024-09-16T23:41:33.450 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-16T23:41:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4500 IOs, 1733996544 bytes 2024-09-16T23:41:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4600 IOs, 1734588416 bytes 2024-09-16T23:41:33.742 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4700 IOs, 1768778752 bytes 2024-09-16T23:41:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4800 IOs, 1822413824 bytes 2024-09-16T23:41:34.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:4900 IOs, 1830887424 bytes 2024-09-16T23:41:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5000 IOs, 1863696384 bytes 2024-09-16T23:41:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5100 IOs, 1941700608 bytes 2024-09-16T23:41:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5200 IOs, 2015944704 bytes 2024-09-16T23:41:34.669 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5300 IOs, 2069017600 bytes 2024-09-16T23:41:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5400 IOs, 2083602432 bytes 2024-09-16T23:41:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5500 IOs, 2152954880 bytes 2024-09-16T23:41:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5600 IOs, 2204050432 bytes 2024-09-16T23:41:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5700 IOs, 2231351296 bytes 2024-09-16T23:41:35.244 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5800 IOs, 2270382080 bytes 2024-09-16T23:41:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:5900 IOs, 2297647104 bytes 2024-09-16T23:41:35.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6000 IOs, 2305208320 bytes 2024-09-16T23:41:35.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6100 IOs, 2358871040 bytes 2024-09-16T23:41:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6200 IOs, 2372242432 bytes 2024-09-16T23:41:35.772 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6300 IOs, 2408294400 bytes 2024-09-16T23:41:35.879 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6400 IOs, 2432630784 bytes 2024-09-16T23:41:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6500 IOs, 2480550912 bytes 2024-09-16T23:41:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6600 IOs, 2497409024 bytes 2024-09-16T23:41:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6700 IOs, 2528361472 bytes 2024-09-16T23:41:36.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6800 IOs, 2580591616 bytes 2024-09-16T23:41:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:6900 IOs, 2623845376 bytes 2024-09-16T23:41:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7000 IOs, 2666035200 bytes 2024-09-16T23:41:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7100 IOs, 2710593536 bytes 2024-09-16T23:41:36.856 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7200 IOs, 2800196608 bytes 2024-09-16T23:41:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7300 IOs, 2846191616 bytes 2024-09-16T23:41:37.110 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7400 IOs, 2892104704 bytes 2024-09-16T23:41:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7500 IOs, 2917729280 bytes 2024-09-16T23:41:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7600 IOs, 2931348480 bytes 2024-09-16T23:41:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7700 IOs, 3012870144 bytes 2024-09-16T23:41:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7800 IOs, 3055102976 bytes 2024-09-16T23:41:37.672 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:7900 IOs, 3115753472 bytes 2024-09-16T23:41:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8000 IOs, 3227991040 bytes 2024-09-16T23:41:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8100 IOs, 3241618432 bytes 2024-09-16T23:41:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8200 IOs, 3311457280 bytes 2024-09-16T23:41:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8300 IOs, 3376130048 bytes 2024-09-16T23:41:38.358 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8400 IOs, 3410356224 bytes 2024-09-16T23:41:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8500 IOs, 3418082304 bytes 2024-09-16T23:41:38.609 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8600 IOs, 3510112256 bytes 2024-09-16T23:41:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8700 IOs, 3542887424 bytes 2024-09-16T23:41:38.835 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8800 IOs, 3579761664 bytes 2024-09-16T23:41:38.943 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:8900 IOs, 3603816448 bytes 2024-09-16T23:41:39.038 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9000 IOs, 3621487616 bytes 2024-09-16T23:41:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:9100 IOs, 3668572160 bytes 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=124 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 124 -eq 124 ']' 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=present 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:41:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:41:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id=14446b014e23 2024-09-16T23:41:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:41:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:41:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_10 present 14446b014e23 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=present 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id=14446b014e23 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:41:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:41:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:41:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id=14446b014e23 2024-09-16T23:41:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 14446b014e23 2024-09-16T23:41:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -z 14446b014e23 -o 14446b014e23 = 14446b014e23 2024-09-16T23:41:39.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_state=present 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test present = present 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-16T23:41:40.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:41:40.174+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556522f59800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-16T23:41:40.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:41:40.174+0000 7f3e05f96640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x556522f59800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=195, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=719c1107-0abe-4e5e-9d82-f041ee6c7a04, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-16T23:41:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:Mirroring disabled 2024-09-16T23:41:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-16T23:41:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:41:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local state=deleted 2024-09-16T23:41:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:41:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local s 2024-09-16T23:41:45.191 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:41:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:41:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T23:41:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ image_id= 2024-09-16T23:41:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-16T23:41:45.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0.1 2024-09-16T23:41:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-16T23:41:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local test_state=deleted 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image_id= 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_state=deleted 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local current_image_id 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ get_image_id cluster1 mirror image_10 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local image=image_10 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-16T23:41:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-16T23:41:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ current_image_id= 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n '' 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test deleted = deleted 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:45.427 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:41:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:41:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:45.428 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:41:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:41:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:45.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:45.521 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 1 2024-09-16T23:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:46.523 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:41:46.578 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:41:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 2 2024-09-16T23:41:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:41:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:41:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:41:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 4 2024-09-16T23:41:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:41:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:41:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:41:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:41:52.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:41:52.791 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:41:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:41:52.846 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:41:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:41:52.849 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:41:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 8 2024-09-16T23:42:00.851 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:42:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:42:00.852 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:42:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:42:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 0% complete...failed. 2024-09-16T23:42:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:rbd: error: image still has watchers 2024-09-16T23:42:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2024-09-16T23:42:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-16T23:42:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 16 2024-09-16T23:42:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ remove_image cluster2 mirror image_10 2024-09-16T23:42:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local image=image_10 2024-09-16T23:42:16.958 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-16T23:42:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-16T23:42:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T23:42:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-16T23:42:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ tee -a /tmp/tmp.JxTgO6em7N/rbd-mirror.test.log 2024-09-16T23:42:17.236 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ date '+%F %T' 2024-09-16T23:42:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo 2024-09-16 23:42:17 TEST: check if removed 'images'\''' OMAP are removed 2024-09-16T23:42:17.237 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16 23:42:17 TEST: check if removed images' OMAP are removed 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirroring 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=status_global 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.255 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=status_global 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T23:42:17.257 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c status_global 2024-09-16T23:42:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirroring 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=image_ 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-16T23:42:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=image_ 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c image_ 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-16T23:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.376 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=image_map 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=image_map 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster1 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c image_map 2024-09-16T23:42:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-16T23:42:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirroring 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=status_global 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-16T23:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=status_global 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T23:42:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c status_global 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirroring 2024-09-16T23:42:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=image_ 2024-09-16T23:42:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=image_ 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-16T23:42:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirroring 2024-09-16T23:42:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-16T23:42:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c image_ 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local pool=mirror 2024-09-16T23:42:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local filter=image_map 2024-09-16T23:42:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-16T23:42:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ sleep 0 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local filter=image_map 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local cluster=cluster2 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local pool=mirror 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ local obj_name=rbd_mirror_leader 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-16T23:42:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ grep -c image_map 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test 0 = 0 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ error_code=0 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set -e 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 0 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ cleanup 0 2024-09-16T23:42:17.518 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local error_code=0 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ set +e 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z '' ']' 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster instance 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:42:17.519 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T23:42:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror' does not exist 2024-09-16T23:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:42:18.750 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T23:42:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror' does not exist 2024-09-16T23:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:42:19.363 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T23:42:19.911 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.913+0000 7f3e12fb0640 -1 rbd::mirror::LeaderWatcher: 0x556521f8cc00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.913+0000 7f3e0cfa4640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55652244d680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.913+0000 7f3e0cfa4640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55652200e240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.913+0000 7f3e0cfa4640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55652200f200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e0c7a3640 -1 rbd::mirror::PoolWatcher: 0x556520e2c960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e0c7a3640 -1 rbd::mirror::PoolWatcher: 0x556520e2d4a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e0cfa4640 -1 rbd::mirror::PoolWatcher: 0x55652252c780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e05f96640 -1 rbd::mirror::PoolWatcher: 0x55652252cd20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e05f96640 -1 rbd::mirror::PoolWatcher: 0x55652252d680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e05f96640 -1 rbd::mirror::PoolWatcher: 0x55652252d4a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e137b1640 -1 rbd::mirror::LeaderWatcher: 0x556521f8cc00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T23:42:19.912 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:19.914+0000 7f3e137b1640 -1 rbd::mirror::LeaderWatcher: 0x556521f8cc00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T23:42:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror_parent' does not exist 2024-09-16T23:42:20.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dca77e640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9a9a900 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T23:42:20.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc3f71640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c6ca77a000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:20.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc3f71640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c6ca0d9d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:20.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc3f71640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c6ca0d8480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc4772640 -1 rbd::mirror::PoolWatcher: 0x55c6ca678d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc3f71640 -1 rbd::mirror::PoolWatcher: 0x55c6c8b44d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dc3f71640 -1 rbd::mirror::PoolWatcher: 0x55c6ca6792c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dbdf65640 -1 rbd::mirror::PoolWatcher: 0x55c6ca469680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.745+0000 7f6dbdf65640 -1 rbd::mirror::PoolWatcher: 0x55c6ca1cab40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.746+0000 7f6dbd764640 -1 rbd::mirror::PoolWatcher: 0x55c6ca18da40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.746+0000 7f6dca77e640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9a9a900 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T23:42:20.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:20.746+0000 7f6dca77e640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9a9a900 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T23:42:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ CEPH_ARGS= 2024-09-16T23:42:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T23:42:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:2024-09-16T23:42:21.032+0000 7f069bfff640 0 --2- 172.21.15.98:0/553604960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f069c0aa040 0x7f069c0acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T23:42:21.728 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:21.730+0000 7f2565e88640 -1 rbd::mirror::LeaderWatcher: 0x560f3ae8f200 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-16T23:42:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:21.730+0000 7f255165f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560f3adc4000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:21.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:21.731+0000 7f2551e60640 -1 rbd::mirror::PoolWatcher: 0x560f3ab30000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:21.732+0000 7f2565e88640 -1 rbd::mirror::LeaderWatcher: 0x560f3ae8f200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T23:42:21.730 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:21.732+0000 7f2565e88640 -1 rbd::mirror::LeaderWatcher: 0x560f3ae8f200 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-16T23:42:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:pool 'mirror_parent' does not exist 2024-09-16T23:42:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T23:42:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirrors cluster1 2024-09-16T23:42:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1 2024-09-16T23:42:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.285 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 0 3 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:0 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster1:1 '' 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:1 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster1:2 '' 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:2 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster1:3 '' 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster1:3 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirrors cluster2 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:++ seq 0 3 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:0 2024-09-16T23:42:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster2:1 '' 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:1 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster2:2 '' 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:2 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.293 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ stop_mirror cluster2:3 '' 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local cluster=cluster2:3 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ local sig= 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test -n 1 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ return 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' -z 1 ']' 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ test '' = /tmp/tmp.JxTgO6em7N 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ rm -Rf /tmp/tmp.JxTgO6em7N 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ echo OK 2024-09-16T23:42:22.294 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stdout:OK 2024-09-16T23:42:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi098.stderr:+ exit 0 2024-09-16T23:42:22.296 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T23:42:22.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-16T23:42:22.416 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-16T23:42:22.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-16T23:42:22.684 DEBUG:teuthology.parallel:result is None 2024-09-16T23:42:22.684 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T23:42:22.712 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T23:42:22.712 DEBUG:teuthology.orchestra.run.smithi098:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T23:42:22.775 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T23:42:22.775 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T23:42:22.785 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T23:42:22.785 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2024-09-16T23:42:22.785 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T23:42:22.785 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-16T23:42:22.884 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-16T23:42:22.884 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 12 2024-09-16T23:42:22.884 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-16T23:42:22.884 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:22.940 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-16T23:42:22.941 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:22.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6dcb780640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 46126) UID: 1000 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6db6f57640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55c6c9c5c240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6dcaf7f640 -1 rbd::mirror::InstanceWatcher: 0x55c6c9db8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6dcaf7f640 -1 rbd::mirror::InstanceWatcher: 0x55c6ca174000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6db6756640 -1 rbd::mirror::PoolWatcher: 0x55c6ca18c000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6dc4772640 -1 rbd::mirror::InstanceWatcher: 0x55c6c9db8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.944+0000 7f6dc4772640 -1 rbd::mirror::InstanceWatcher: 0x55c6ca174000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:22.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6daff49640 -1 rbd::mirror::PoolWatcher: 0x55c6ca18d680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dbdf65640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9d0e5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dbdf65640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9d0e360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0c1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dc3f71640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9d0e000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dc3f71640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9d0e120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0c1c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dca77e640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9dd0f00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dca77e640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9dd0f00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.945+0000 7f6dcaf7f640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9dd0f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:22.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6dcaf7f640 -1 rbd::mirror::InstanceWatcher: 0x55c6c9db9880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4861 object: (2) No such file or directory 2024-09-16T23:42:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6db6756640 -1 rbd::mirror::InstanceWatcher: 0x55c6c9db9880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6db074a640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9f465a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0ca80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6db6f57640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6c9f03680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.946+0000 7f6dca77e640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c9d0ca80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.948+0000 7f6dcaf7f640 -1 rbd::mirror::LeaderWatcher: 0x55c6c9a9a900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:22.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dcaf7f640 -1 rbd::mirror::InstanceWatcher: 0x55c6ca521c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4858 object: (2) No such file or directory 2024-09-16T23:42:22.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dc4772640 -1 rbd::mirror::InstanceWatcher: 0x55c6ca521c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:22.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dbd764640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6ca6c27e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dca77e640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c8b7ac40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dc3f71640 -1 rbd::mirror::MirrorStatusUpdater 0x55c6ca6a1320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:22.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi098.stderr:2024-09-16T23:42:22.949+0000 7f6dcaf7f640 -1 rbd::mirror::NamespaceReplayer: 0x55c6c8b7ac40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:22.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-16T23:42:22.960 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.074 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-16T23:42:23.074 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.077+0000 7f33bc249640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 72326) UID: 1000 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33bb247640 -1 rbd::mirror::LeaderWatcher: 0x559503d87200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33bba48640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6871 object: (2) No such file or directory 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33bba48640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6871 object: (2) No such file or directory 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33bba48640 -1 rbd::mirror::InstanceWatcher: 0x559504068000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6873 object: (2) No such file or directory 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33b523b640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.076 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33b523b640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33a721f640 -1 rbd::mirror::InstanceWatcher: 0x559504068000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33a7a20640 -1 rbd::mirror::MirrorStatusUpdater 0x559503dd1b00 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.078+0000 7f33bb247640 -1 rbd::mirror::NamespaceReplayer: 0x559503bfca80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.077 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33ae22d640 -1 rbd::mirror::MirrorStatusUpdater 0x559503bf65a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33ae22d640 -1 rbd::mirror::MirrorStatusUpdater 0x559503bf6360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33bb247640 -1 rbd::mirror::NamespaceReplayer: 0x559502c97dc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33bb247640 -1 rbd::mirror::NamespaceReplayer: 0x559503bfc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33b4a3a640 -1 rbd::mirror::MirrorStatusUpdater 0x559503bf6000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33b4a3a640 -1 rbd::mirror::MirrorStatusUpdater 0x559503bf6120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33bba48640 -1 rbd::mirror::NamespaceReplayer: 0x559502c97dc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.079+0000 7f33bba48640 -1 rbd::mirror::NamespaceReplayer: 0x559503bfc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.080+0000 7f33bba48640 -1 rbd::mirror::LeaderWatcher: 0x5595049b8c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.081+0000 7f33bba48640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6871 object: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.081+0000 7f33b523b640 -1 rbd::mirror::InstanceWatcher: 0x559503b7c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.082+0000 7f33aea2e640 -1 rbd::mirror::MirrorStatusUpdater 0x5595048185a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.082+0000 7f33bba48640 -1 rbd::mirror::NamespaceReplayer: 0x559502c96000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.082+0000 7f33b4a3a640 -1 rbd::mirror::MirrorStatusUpdater 0x5595047e5560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.082+0000 7f33bb247640 -1 rbd::mirror::NamespaceReplayer: 0x559502c96000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-16T23:42:23.091 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.175 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-16T23:42:23.175 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.178+0000 7fb47bf2e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 67969) UID: 1000 2024-09-16T23:42:23.176 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.178+0000 7fb47af2c640 -1 rbd::mirror::LeaderWatcher: 0x55cfc5369200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.179+0000 7fb47b72d640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6455 object: (2) No such file or directory 2024-09-16T23:42:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.179+0000 7fb47b72d640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6455 object: (2) No such file or directory 2024-09-16T23:42:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.179+0000 7fb47b72d640 -1 rbd::mirror::InstanceWatcher: 0x55cfc562c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6457 object: (2) No such file or directory 2024-09-16T23:42:23.177 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.179+0000 7fb466f04640 -1 rbd::mirror::InstanceWatcher: 0x55cfc562c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.180+0000 7fb467705640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc53d59e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.180+0000 7fb47af2c640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc51d0a80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.180+0000 7fb474f20640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.178 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.180+0000 7fb474f20640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb46e713640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc51ce5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb47b72d640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc3fc1dc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb46df12640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc51ce360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb47af2c640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc51d0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb47471f640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc51ce000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb47b72d640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc3fc1dc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb474f20640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc51ce120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.179 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.181+0000 7fb47af2c640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc51d0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.182+0000 7fb47af2c640 -1 rbd::mirror::LeaderWatcher: 0x55cfc50b0c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.182+0000 7fb47af2c640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6455 object: (2) No such file or directory 2024-09-16T23:42:23.180 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.183+0000 7fb47471f640 -1 rbd::mirror::InstanceWatcher: 0x55cfc5158000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.183+0000 7fb46e713640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc4ed27e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.183+0000 7fb47af2c640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc3fc0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.183+0000 7fb474f20640 -1 rbd::mirror::MirrorStatusUpdater 0x55cfc4ec1320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.183+0000 7fb47b72d640 -1 rbd::mirror::NamespaceReplayer: 0x55cfc3fc0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.192 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-16T23:42:23.192 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.279 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-16T23:42:23.279 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a8fab640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 71289) UID: 1000 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a7fa9640 -1 rbd::mirror::LeaderWatcher: 0x55655a5f1200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a87aa640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6766 object: (2) No such file or directory 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a87aa640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6766 object: (2) No such file or directory 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a87aa640 -1 rbd::mirror::InstanceWatcher: 0x55655a8aa000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6768 object: (2) No such file or directory 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a1f9d640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.283+0000 7fa9a179c640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa994782640 -1 rbd::mirror::InstanceWatcher: 0x55655a8aa000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa99af8f640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a42ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa99af8f640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a42c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a7fa9640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a7fa9640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43e1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa98d774640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a6267e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a87aa640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43ea80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a1f9d640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a42c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a1f9d640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a42c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a87aa640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a87aa640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43e1c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa993f81640 -1 rbd::mirror::MirrorStatusUpdater 0x55655a60b9e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.284+0000 7fa9a7fa9640 -1 rbd::mirror::NamespaceReplayer: 0x55655a43ea80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.286+0000 7fa9a7fa9640 -1 rbd::mirror::LeaderWatcher: 0x55655a326c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.286+0000 7fa9a7fa9640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6766 object: (2) No such file or directory 2024-09-16T23:42:23.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.287+0000 7fa9a179c640 -1 rbd::mirror::InstanceWatcher: 0x55655a3b2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.287+0000 7fa99b790640 -1 rbd::mirror::MirrorStatusUpdater 0x55655ae2c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.287+0000 7fa9a7fa9640 -1 rbd::mirror::NamespaceReplayer: 0x55655928bdc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.287+0000 7fa9a1f9d640 -1 rbd::mirror::MirrorStatusUpdater 0x55655ada9440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.287+0000 7fa9a87aa640 -1 rbd::mirror::NamespaceReplayer: 0x55655928bdc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.296 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-16T23:42:23.296 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.352 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-16T23:42:23.352 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.355+0000 7f3e13fb2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 55608) UID: 1000 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e12fb0640 -1 rbd::mirror::LeaderWatcher: 0x556522247200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e137b1640 -1 rbd::mirror::InstanceWatcher: 0x55652202c380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5188 object: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e137b1640 -1 rbd::mirror::InstanceWatcher: 0x55652202c700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5188 object: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e137b1640 -1 rbd::mirror::InstanceWatcher: 0x5565224fe000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5190 object: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e0cfa4640 -1 rbd::mirror::InstanceWatcher: 0x55652202c380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.356+0000 7f3e0cfa4640 -1 rbd::mirror::InstanceWatcher: 0x55652202c700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3dff789640 -1 rbd::mirror::InstanceWatcher: 0x5565224fe000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e05f96640 -1 rbd::mirror::MirrorStatusUpdater 0x556522096360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e05f96640 -1 rbd::mirror::MirrorStatusUpdater 0x5565220965a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e137b1640 -1 rbd::mirror::NamespaceReplayer: 0x556522094000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e137b1640 -1 rbd::mirror::NamespaceReplayer: 0x5565220941c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e0cfa4640 -1 rbd::mirror::MirrorStatusUpdater 0x556522096000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3df877b640 -1 rbd::mirror::MirrorStatusUpdater 0x556522096b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e0cfa4640 -1 rbd::mirror::MirrorStatusUpdater 0x556522096120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e12fb0640 -1 rbd::mirror::NamespaceReplayer: 0x556522094000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e12fb0640 -1 rbd::mirror::NamespaceReplayer: 0x556522094e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e12fb0640 -1 rbd::mirror::NamespaceReplayer: 0x5565220941c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3dfef88640 -1 rbd::mirror::MirrorStatusUpdater 0x55652224dd40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.356 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.357+0000 7f3e137b1640 -1 rbd::mirror::NamespaceReplayer: 0x556522094e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.359+0000 7f3e137b1640 -1 rbd::mirror::LeaderWatcher: 0x556521f8cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e137b1640 -1 rbd::mirror::InstanceWatcher: 0x55652202c000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5188 object: (2) No such file or directory 2024-09-16T23:42:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e0c7a3640 -1 rbd::mirror::InstanceWatcher: 0x55652202c000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e06797640 -1 rbd::mirror::MirrorStatusUpdater 0x556521da47e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e137b1640 -1 rbd::mirror::NamespaceReplayer: 0x556520e63dc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e0cfa4640 -1 rbd::mirror::MirrorStatusUpdater 0x556521d79440 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi098.stderr:2024-09-16T23:42:23.360+0000 7f3e12fb0640 -1 rbd::mirror::NamespaceReplayer: 0x556520e63dc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-16T23:42:23.372 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.457 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-16T23:42:23.458 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.458 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.461+0000 7f1dfc017640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 63914) UID: 1000 2024-09-16T23:42:23.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.461+0000 7f1dfb816640 -1 rbd::mirror::LeaderWatcher: 0x563a3ead9200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dfb816640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5682 object: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dfb816640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5682 object: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dfb015640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5684 object: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1df4808640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1df5009640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1de6fed640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dee7fc640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e936480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dee7fc640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e936360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dfb816640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e934000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.462+0000 7f1dfb816640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e9341c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1df4808640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e936000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1de0fe1640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e936b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1df4808640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e936120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1dfb015640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e934000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1dfb015640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e934c40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1dfb015640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e9341c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1de77ee640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3eaefc20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.463+0000 7f1dfb816640 -1 rbd::mirror::NamespaceReplayer: 0x563a3e934c40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.465+0000 7f1dfb816640 -1 rbd::mirror::LeaderWatcher: 0x563a3e82ac00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.465+0000 7f1dfb816640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5682 object: (2) No such file or directory 2024-09-16T23:42:23.463 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.465+0000 7f1df5009640 -1 rbd::mirror::InstanceWatcher: 0x563a3e8c4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.466+0000 7f1dedffb640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e6826c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.466+0000 7f1dfb816640 -1 rbd::mirror::NamespaceReplayer: 0x563a3d727dc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.466+0000 7f1df4808640 -1 rbd::mirror::MirrorStatusUpdater 0x563a3e629320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi098.stderr:2024-09-16T23:42:23.466+0000 7f1dfb015640 -1 rbd::mirror::NamespaceReplayer: 0x563a3d727dc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-16T23:42:23.474 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.514 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-16T23:42:23.515 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.518+0000 7f2566689640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 58567) UID: 1000 2024-09-16T23:42:23.516 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f2565e88640 -1 rbd::mirror::LeaderWatcher: 0x560f3ae8f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f2565687640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5394 object: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f2565687640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5394 object: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f2565e88640 -1 rbd::mirror::InstanceWatcher: 0x560f3b166000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5396 object: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f255f67b640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f255ee7a640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.519+0000 7f255165f640 -1 rbd::mirror::InstanceWatcher: 0x560f3b166000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2551e60640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aea99e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2565687640 -1 rbd::mirror::NamespaceReplayer: 0x560f3ae58540 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2558e6e640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3acf4360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2558e6e640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3acf45a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2565687640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d96e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2565687640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d96000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f255f67b640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3acf4000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f255f67b640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3acf4120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2565e88640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d96e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.520+0000 7f2565e88640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d96000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.519 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.522+0000 7f2565e88640 -1 rbd::mirror::LeaderWatcher: 0x560f3bab0c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.522+0000 7f2565e88640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5394 object: (2) No such file or directory 2024-09-16T23:42:23.520 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.522+0000 7f255ee7a640 -1 rbd::mirror::InstanceWatcher: 0x560f3ac86000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.523+0000 7f255866d640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3b9245a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.523+0000 7f2565e88640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d968c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.523+0000 7f255f67b640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3b8bf560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.521 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi098.stderr:2024-09-16T23:42:23.523+0000 7f2565687640 -1 rbd::mirror::NamespaceReplayer: 0x560f39d968c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-16T23:42:23.532 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-16T23:42:23.542 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-16T23:42:23.542 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:23.573 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.576+0000 7fe6a538c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 71671) UID: 1000 2024-09-16T23:42:23.575 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe6a438a640 -1 rbd::mirror::InstanceWatcher: 0x55867e402380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe6a4b8b640 -1 rbd::mirror::InstanceWatcher: 0x55867e402700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe690b63640 -1 rbd::mirror::InstanceWatcher: 0x55867e8c6000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe69e37e640 -1 rbd::mirror::InstanceWatcher: 0x55867e402380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe69db7d640 -1 rbd::mirror::InstanceWatcher: 0x55867e402700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe68a356640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e40ab40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe6a4b8b640 -1 rbd::mirror::NamespaceReplayer: 0x55867e408a80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe697370640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e40a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe697370640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e40a5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe6a438a640 -1 rbd::mirror::NamespaceReplayer: 0x55867e408000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.577+0000 7fe6a438a640 -1 rbd::mirror::NamespaceReplayer: 0x55867e4081c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.578+0000 7fe69e37e640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e40a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.576 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.578+0000 7fe6a4b8b640 -1 rbd::mirror::NamespaceReplayer: 0x55867e408000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.578+0000 7fe69db7d640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e40a120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.578+0000 7fe6a438a640 -1 rbd::mirror::NamespaceReplayer: 0x55867e4081c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.577 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.579+0000 7fe6a438a640 -1 rbd::mirror::LeaderWatcher: 0x55867e364c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe6a438a640 -1 rbd::mirror::InstanceWatcher: 0x55867e402000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6093 object: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe69e37e640 -1 rbd::mirror::InstanceWatcher: 0x55867e402000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe697b71640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e1b26c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe6a4b8b640 -1 rbd::mirror::NamespaceReplayer: 0x55867d24fdc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe69db7d640 -1 rbd::mirror::MirrorStatusUpdater 0x55867e14d320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-16T23:42:23.578 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi098.stderr:2024-09-16T23:42:23.580+0000 7fe6a438a640 -1 rbd::mirror::NamespaceReplayer: 0x55867d24fdc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-16T23:42:23.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-16T23:42:23.589 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T23:42:23.600 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T23:42:23.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:42:23.916 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:23.917 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:42:23.928 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-16T23:42:22.803780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392796,"kb_used_data":9444,"kb_used_omap":4,"kb_used_meta":4383291,"kb_avail":278722724,"statfs":{"total":289910292480,"available":285412069376,"internally_reserved":0,"allocated":9670656,"data_stored":6286888,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4488490319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536755254,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113910,"num_read_kb":-39649078,"num_write":-645978,"num_write_kb":-241835448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-25165824,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161929+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:42:19.161929+0000","last_peered":"2024-09-16T23:42:19.161929+0000","last_clean":"2024-09-16T23:42:19.161929+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:42:19.161929+0000","last_undegraded":"2024-09-16T23:42:19.161929+0000","last_fullsized":"2024-09-16T23:42:19.161929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161942+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:42:19.161942+0000","last_peered":"2024-09-16T23:42:19.161942+0000","last_clean":"2024-09-16T23:42:19.161942+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:42:19.161942+0000","last_undegraded":"2024-09-16T23:42:19.161942+0000","last_fullsized":"2024-09-16T23:42:19.161942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159791+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:42:19.159791+0000","last_peered":"2024-09-16T23:42:19.159791+0000","last_clean":"2024-09-16T23:42:19.159791+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:42:19.159791+0000","last_undegraded":"2024-09-16T23:42:19.159791+0000","last_fullsized":"2024-09-16T23:42:19.159791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.155471+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:42:19.155471+0000","last_peered":"2024-09-16T23:42:19.155471+0000","last_clean":"2024-09-16T23:42:19.155471+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:42:19.155471+0000","last_undegraded":"2024-09-16T23:42:19.155471+0000","last_fullsized":"2024-09-16T23:42:19.155471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.074730+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:42:20.074730+0000","last_peered":"2024-09-16T23:42:20.074730+0000","last_clean":"2024-09-16T23:42:20.074730+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:42:20.074730+0000","last_undegraded":"2024-09-16T23:42:20.074730+0000","last_fullsized":"2024-09-16T23:42:20.074730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.184583+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:42:19.184583+0000","last_peered":"2024-09-16T23:42:19.184583+0000","last_clean":"2024-09-16T23:42:19.184583+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:42:19.184583+0000","last_undegraded":"2024-09-16T23:42:19.184583+0000","last_fullsized":"2024-09-16T23:42:19.184583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.188005+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:42:19.188005+0000","last_peered":"2024-09-16T23:42:19.188005+0000","last_clean":"2024-09-16T23:42:19.188005+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:42:19.188005+0000","last_undegraded":"2024-09-16T23:42:19.188005+0000","last_fullsized":"2024-09-16T23:42:19.188005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":780,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159128+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:42:19.159128+0000","last_peered":"2024-09-16T23:42:19.159128+0000","last_clean":"2024-09-16T23:42:19.159128+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:42:19.159128+0000","last_undegraded":"2024-09-16T23:42:19.159128+0000","last_fullsized":"2024-09-16T23:42:19.159128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":847,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.158462+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:42:19.158462+0000","last_peered":"2024-09-16T23:42:19.158462+0000","last_clean":"2024-09-16T23:42:19.158462+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:42:19.158462+0000","last_undegraded":"2024-09-16T23:42:19.158462+0000","last_fullsized":"2024-09-16T23:42:19.158462+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361276,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1358142,"kb_avail":93010564,"statfs":{"total":96636764160,"available":95242817536,"internally_reserved":0,"allocated":3186688,"data_stored":1703406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1390737861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429432,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1426238,"kb_avail":92942408,"statfs":{"total":96636764160,"available":95173025792,"internally_reserved":0,"allocated":3248128,"data_stored":2293793,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1460468165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602088,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":1598910,"kb_avail":92769752,"statfs":{"total":96636764160,"available":94996226048,"internally_reserved":0,"allocated":3235840,"data_stored":2289689,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1637284293},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:23.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:42:24.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:24.211 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:42:24.223 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-16T23:42:22.803780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392796,"kb_used_data":9444,"kb_used_omap":4,"kb_used_meta":4383291,"kb_avail":278722724,"statfs":{"total":289910292480,"available":285412069376,"internally_reserved":0,"allocated":9670656,"data_stored":6286888,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4488490319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536755254,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113910,"num_read_kb":-39649078,"num_write":-645978,"num_write_kb":-241835448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-25165824,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161929+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:42:19.161929+0000","last_peered":"2024-09-16T23:42:19.161929+0000","last_clean":"2024-09-16T23:42:19.161929+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:42:19.161929+0000","last_undegraded":"2024-09-16T23:42:19.161929+0000","last_fullsized":"2024-09-16T23:42:19.161929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161942+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:42:19.161942+0000","last_peered":"2024-09-16T23:42:19.161942+0000","last_clean":"2024-09-16T23:42:19.161942+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:42:19.161942+0000","last_undegraded":"2024-09-16T23:42:19.161942+0000","last_fullsized":"2024-09-16T23:42:19.161942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159791+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:42:19.159791+0000","last_peered":"2024-09-16T23:42:19.159791+0000","last_clean":"2024-09-16T23:42:19.159791+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:42:19.159791+0000","last_undegraded":"2024-09-16T23:42:19.159791+0000","last_fullsized":"2024-09-16T23:42:19.159791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.155471+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:42:19.155471+0000","last_peered":"2024-09-16T23:42:19.155471+0000","last_clean":"2024-09-16T23:42:19.155471+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:42:19.155471+0000","last_undegraded":"2024-09-16T23:42:19.155471+0000","last_fullsized":"2024-09-16T23:42:19.155471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.074730+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:42:20.074730+0000","last_peered":"2024-09-16T23:42:20.074730+0000","last_clean":"2024-09-16T23:42:20.074730+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:42:20.074730+0000","last_undegraded":"2024-09-16T23:42:20.074730+0000","last_fullsized":"2024-09-16T23:42:20.074730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.184583+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:42:19.184583+0000","last_peered":"2024-09-16T23:42:19.184583+0000","last_clean":"2024-09-16T23:42:19.184583+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:42:19.184583+0000","last_undegraded":"2024-09-16T23:42:19.184583+0000","last_fullsized":"2024-09-16T23:42:19.184583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.188005+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:42:19.188005+0000","last_peered":"2024-09-16T23:42:19.188005+0000","last_clean":"2024-09-16T23:42:19.188005+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:42:19.188005+0000","last_undegraded":"2024-09-16T23:42:19.188005+0000","last_fullsized":"2024-09-16T23:42:19.188005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":780,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159128+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:42:19.159128+0000","last_peered":"2024-09-16T23:42:19.159128+0000","last_clean":"2024-09-16T23:42:19.159128+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:42:19.159128+0000","last_undegraded":"2024-09-16T23:42:19.159128+0000","last_fullsized":"2024-09-16T23:42:19.159128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":847,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.158462+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:42:19.158462+0000","last_peered":"2024-09-16T23:42:19.158462+0000","last_clean":"2024-09-16T23:42:19.158462+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:42:19.158462+0000","last_undegraded":"2024-09-16T23:42:19.158462+0000","last_fullsized":"2024-09-16T23:42:19.158462+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361276,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1358142,"kb_avail":93010564,"statfs":{"total":96636764160,"available":95242817536,"internally_reserved":0,"allocated":3186688,"data_stored":1703406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1390737861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429432,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1426238,"kb_avail":92942408,"statfs":{"total":96636764160,"available":95173025792,"internally_reserved":0,"allocated":3248128,"data_stored":2293793,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1460468165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602088,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":1598910,"kb_avail":92769752,"statfs":{"total":96636764160,"available":94996226048,"internally_reserved":0,"allocated":3235840,"data_stored":2289689,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1637284293},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:24.224 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T23:42:24.224 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:42:24.504 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:24.504 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:42:24.516 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-16T23:42:22.803780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4392796,"kb_used_data":9444,"kb_used_omap":4,"kb_used_meta":4383291,"kb_avail":278722724,"statfs":{"total":289910292480,"available":285412069376,"internally_reserved":0,"allocated":9670656,"data_stored":6286888,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4488490319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536755254,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113910,"num_read_kb":-39649078,"num_write":-645978,"num_write_kb":-241835448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-25165824,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161929+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:42:19.161929+0000","last_peered":"2024-09-16T23:42:19.161929+0000","last_clean":"2024-09-16T23:42:19.161929+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:42:19.161929+0000","last_undegraded":"2024-09-16T23:42:19.161929+0000","last_fullsized":"2024-09-16T23:42:19.161929+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":778,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.161942+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:42:19.161942+0000","last_peered":"2024-09-16T23:42:19.161942+0000","last_clean":"2024-09-16T23:42:19.161942+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:42:19.161942+0000","last_undegraded":"2024-09-16T23:42:19.161942+0000","last_fullsized":"2024-09-16T23:42:19.161942+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159791+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:42:19.159791+0000","last_peered":"2024-09-16T23:42:19.159791+0000","last_clean":"2024-09-16T23:42:19.159791+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:42:19.159791+0000","last_undegraded":"2024-09-16T23:42:19.159791+0000","last_fullsized":"2024-09-16T23:42:19.159791+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.155471+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:42:19.155471+0000","last_peered":"2024-09-16T23:42:19.155471+0000","last_clean":"2024-09-16T23:42:19.155471+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:42:19.155471+0000","last_undegraded":"2024-09-16T23:42:19.155471+0000","last_fullsized":"2024-09-16T23:42:19.155471+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":782,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.074730+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:42:20.074730+0000","last_peered":"2024-09-16T23:42:20.074730+0000","last_clean":"2024-09-16T23:42:20.074730+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:42:20.074730+0000","last_undegraded":"2024-09-16T23:42:20.074730+0000","last_fullsized":"2024-09-16T23:42:20.074730+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.184583+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:42:19.184583+0000","last_peered":"2024-09-16T23:42:19.184583+0000","last_clean":"2024-09-16T23:42:19.184583+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:42:19.184583+0000","last_undegraded":"2024-09-16T23:42:19.184583+0000","last_fullsized":"2024-09-16T23:42:19.184583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":779,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.188005+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:42:19.188005+0000","last_peered":"2024-09-16T23:42:19.188005+0000","last_clean":"2024-09-16T23:42:19.188005+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:42:19.188005+0000","last_undegraded":"2024-09-16T23:42:19.188005+0000","last_fullsized":"2024-09-16T23:42:19.188005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":780,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.159128+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:42:19.159128+0000","last_peered":"2024-09-16T23:42:19.159128+0000","last_clean":"2024-09-16T23:42:19.159128+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:42:19.159128+0000","last_undegraded":"2024-09-16T23:42:19.159128+0000","last_fullsized":"2024-09-16T23:42:19.159128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":847,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-09-16T23:42:19.158462+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:42:19.158462+0000","last_peered":"2024-09-16T23:42:19.158462+0000","last_clean":"2024-09-16T23:42:19.158462+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:42:19.158462+0000","last_undegraded":"2024-09-16T23:42:19.158462+0000","last_fullsized":"2024-09-16T23:42:19.158462+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738805,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361276,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":1358142,"kb_avail":93010564,"statfs":{"total":96636764160,"available":95242817536,"internally_reserved":0,"allocated":3186688,"data_stored":1703406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1390737861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429432,"kb_used_data":3172,"kb_used_omap":1,"kb_used_meta":1426238,"kb_avail":92942408,"statfs":{"total":96636764160,"available":95173025792,"internally_reserved":0,"allocated":3248128,"data_stored":2293793,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1460468165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1602088,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":1598910,"kb_avail":92769752,"statfs":{"total":96636764160,"available":94996226048,"internally_reserved":0,"allocated":3235840,"data_stored":2289689,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1637284293},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:24.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T23:42:24.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":388,"fsid":"9e3c1a33-dc76-489c-9850-c3603aaaec59","created":"2024-09-16T23:06:14.848015+0000","modified":"2024-09-16T23:42:22.153599+0000","last_up_change":"2024-09-16T23:06:24.319317+0000","last_in_change":"2024-09-16T23:06:15.676979+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:06:24.460897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:06:29.974235+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ba21fe61-6dc5-4802-90c4-e76f4cd45d47","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6801","nonce":1900552820}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6803","nonce":1900552820}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6807","nonce":1900552820}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1900552820},{"type":"v1","addr":"172.21.15.98:6805","nonce":1900552820}]},"public_addr":"172.21.15.98:6801/1900552820","cluster_addr":"172.21.15.98:6803/1900552820","heartbeat_back_addr":"172.21.15.98:6807/1900552820","heartbeat_front_addr":"172.21.15.98:6805/1900552820","state":["exists","up"]},{"osd":1,"uuid":"1427ee21-632b-49e9-9047-c174862d612f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6817","nonce":2210406205}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6819","nonce":2210406205}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6823","nonce":2210406205}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":2210406205},{"type":"v1","addr":"172.21.15.98:6821","nonce":2210406205}]},"public_addr":"172.21.15.98:6817/2210406205","cluster_addr":"172.21.15.98:6819/2210406205","heartbeat_back_addr":"172.21.15.98:6823/2210406205","heartbeat_front_addr":"172.21.15.98:6821/2210406205","state":["exists","up"]},{"osd":2,"uuid":"69605198-dc7b-4b62-b637-e200eff28561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6809","nonce":1071820272}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6811","nonce":1071820272}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6815","nonce":1071820272}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1071820272},{"type":"v1","addr":"172.21.15.98:6813","nonce":1071820272}]},"public_addr":"172.21.15.98:6809/1071820272","cluster_addr":"172.21.15.98:6811/1071820272","heartbeat_back_addr":"172.21.15.98:6815/1071820272","heartbeat_front_addr":"172.21.15.98:6813/1071820272","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:21.951674+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.170831+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:06:22.115944+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:42:25.816 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-16T23:42:25.816 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:25.951 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T23:42:25.951 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:25.951 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T23:42:25.961 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-16T23:42:26.247 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T23:42:26.261 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-16T23:42:26.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:26.382 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T23:42:26.382 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:26.382 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T23:42:26.393 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-16T23:42:26.675 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T23:42:26.689 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-16T23:42:26.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:26.811 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-09-16T23:42:26.811 INFO:teuthology.orchestra.run.smithi098.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:26.811 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-09-16T23:42:26.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-16T23:42:27.105 INFO:teuthology.orchestra.run.smithi098.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T23:42:27.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:42:27.398 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:27.399 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:42:27.411 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1467,"stamp":"2024-09-16T23:42:26.804598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":59,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4433092,"kb_used_data":9480,"kb_used_omap":4,"kb_used_meta":4423547,"kb_avail":278682428,"statfs":{"total":289910292480,"available":285370806272,"internally_reserved":0,"allocated":9707520,"data_stored":6301825,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4529712463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536755254,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113919,"num_read_kb":-39649087,"num_write":-645980,"num_write_kb":-241835450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-25165824,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.680923+0000","last_change":"2024-09-16T23:06:33.349345+0000","last_active":"2024-09-16T23:42:22.680923+0000","last_peered":"2024-09-16T23:42:22.680923+0000","last_clean":"2024-09-16T23:42:22.680923+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:42:22.680923+0000","last_undegraded":"2024-09-16T23:42:22.680923+0000","last_fullsized":"2024-09-16T23:42:22.680923+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:33:57.417593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":780,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.683722+0000","last_change":"2024-09-16T23:06:33.349049+0000","last_active":"2024-09-16T23:42:22.683722+0000","last_peered":"2024-09-16T23:42:22.683722+0000","last_clean":"2024-09-16T23:42:22.683722+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:42:22.683722+0000","last_undegraded":"2024-09-16T23:42:22.683722+0000","last_fullsized":"2024-09-16T23:42:22.683722+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:54:00.830566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.682329+0000","last_change":"2024-09-16T23:06:33.349406+0000","last_active":"2024-09-16T23:42:22.682329+0000","last_peered":"2024-09-16T23:42:22.682329+0000","last_clean":"2024-09-16T23:42:22.682329+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:42:22.682329+0000","last_undegraded":"2024-09-16T23:42:22.682329+0000","last_fullsized":"2024-09-16T23:42:22.682329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:57:11.321947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.684056+0000","last_change":"2024-09-16T23:06:33.349591+0000","last_active":"2024-09-16T23:42:22.684056+0000","last_peered":"2024-09-16T23:42:22.684056+0000","last_clean":"2024-09-16T23:42:22.684056+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:42:22.684056+0000","last_undegraded":"2024-09-16T23:42:22.684056+0000","last_fullsized":"2024-09-16T23:42:22.684056+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:49:27.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":784,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.683957+0000","last_change":"2024-09-16T23:06:33.345859+0000","last_active":"2024-09-16T23:42:22.683957+0000","last_peered":"2024-09-16T23:42:22.683957+0000","last_clean":"2024-09-16T23:42:22.683957+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:42:22.683957+0000","last_undegraded":"2024-09-16T23:42:22.683957+0000","last_fullsized":"2024-09-16T23:42:22.683957+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:15:31.417942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.164785+0000","last_change":"2024-09-16T23:06:33.832856+0000","last_active":"2024-09-16T23:42:22.164785+0000","last_peered":"2024-09-16T23:42:22.164785+0000","last_clean":"2024-09-16T23:42:22.164785+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:42:22.164785+0000","last_undegraded":"2024-09-16T23:42:22.164785+0000","last_fullsized":"2024-09-16T23:42:22.164785+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:35:52.432967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":781,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.166437+0000","last_change":"2024-09-16T23:06:33.832838+0000","last_active":"2024-09-16T23:42:22.166437+0000","last_peered":"2024-09-16T23:42:22.166437+0000","last_clean":"2024-09-16T23:42:22.166437+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:42:22.166437+0000","last_undegraded":"2024-09-16T23:42:22.166437+0000","last_fullsized":"2024-09-16T23:42:22.166437+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:32:38.561732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":782,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.680692+0000","last_change":"2024-09-16T23:06:33.349097+0000","last_active":"2024-09-16T23:42:22.680692+0000","last_peered":"2024-09-16T23:42:22.680692+0000","last_clean":"2024-09-16T23:42:22.680692+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:42:22.680692+0000","last_undegraded":"2024-09-16T23:42:22.680692+0000","last_fullsized":"2024-09-16T23:42:22.680692+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:30.331756+0000","last_clean_scrub_stamp":"2024-09-16T23:06:30.331756+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:20:54.978646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":849,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.682573+0000","last_change":"2024-09-16T23:06:26.331949+0000","last_active":"2024-09-16T23:42:22.682573+0000","last_peered":"2024-09-16T23:42:22.682573+0000","last_clean":"2024-09-16T23:42:22.682573+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:42:22.682573+0000","last_undegraded":"2024-09-16T23:42:22.682573+0000","last_fullsized":"2024-09-16T23:42:22.682573+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:06:25.320239+0000","last_clean_scrub_stamp":"2024-09-16T23:06:25.320239+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:14:18.462701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361284,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":1358142,"kb_avail":93010556,"statfs":{"total":96636764160,"available":95242809344,"internally_reserved":0,"allocated":3194880,"data_stored":1707017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1390737861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429440,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":1426238,"kb_avail":92942400,"statfs":{"total":96636764160,"available":95173017600,"internally_reserved":0,"allocated":3256320,"data_stored":2297404,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1460468165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738805,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1642368,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":1639166,"kb_avail":92729472,"statfs":{"total":96636764160,"available":94954979328,"internally_reserved":0,"allocated":3256320,"data_stored":2297404,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1678506437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.415 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T23:06:30.331756+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=30, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.416 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T23:06:25.320239+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=6, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:27.416 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T23:42:47.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T23:42:47.710 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-09-16T23:42:47.711 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-09-16T23:42:47.722 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":1477,"stamp":"2024-09-16T23:42:46.808410+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4433080,"kb_used_data":9480,"kb_used_omap":4,"kb_used_meta":4423547,"kb_avail":278682440,"statfs":{"total":289910292480,"available":285370818560,"internally_reserved":0,"allocated":9707520,"data_stored":6301825,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":4529712463},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002477"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":788,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:32.651521+0000","last_change":"2024-09-16T23:42:32.651521+0000","last_active":"2024-09-16T23:42:32.651521+0000","last_peered":"2024-09-16T23:42:32.651521+0000","last_clean":"2024-09-16T23:42:32.651521+0000","last_became_active":"2024-09-16T23:06:31.342946+0000","last_became_peered":"2024-09-16T23:06:31.342946+0000","last_unstale":"2024-09-16T23:42:32.651521+0000","last_undegraded":"2024-09-16T23:42:32.651521+0000","last_fullsized":"2024-09-16T23:42:32.651521+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:32.651434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:32.651434+0000","last_clean_scrub_stamp":"2024-09-16T23:42:32.651434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:31.111769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086477099999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":788,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:30.659385+0000","last_change":"2024-09-16T23:42:30.659385+0000","last_active":"2024-09-16T23:42:30.659385+0000","last_peered":"2024-09-16T23:42:30.659385+0000","last_clean":"2024-09-16T23:42:30.659385+0000","last_became_active":"2024-09-16T23:06:31.343031+0000","last_became_peered":"2024-09-16T23:06:31.343031+0000","last_unstale":"2024-09-16T23:42:30.659385+0000","last_undegraded":"2024-09-16T23:42:30.659385+0000","last_fullsized":"2024-09-16T23:42:30.659385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:30.659289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:30.659289+0000","last_clean_scrub_stamp":"2024-09-16T23:42:30.659289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:23:20.530078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065355800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":789,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:31.669287+0000","last_change":"2024-09-16T23:42:31.669287+0000","last_active":"2024-09-16T23:42:31.669287+0000","last_peered":"2024-09-16T23:42:31.669287+0000","last_clean":"2024-09-16T23:42:31.669287+0000","last_became_active":"2024-09-16T23:06:31.343023+0000","last_became_peered":"2024-09-16T23:06:31.343023+0000","last_unstale":"2024-09-16T23:42:31.669287+0000","last_undegraded":"2024-09-16T23:42:31.669287+0000","last_fullsized":"2024-09-16T23:42:31.669287+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:31.669193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:31.669193+0000","last_clean_scrub_stamp":"2024-09-16T23:42:31.669193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:00:16.313893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091879099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":789,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:27.680721+0000","last_change":"2024-09-16T23:42:27.680721+0000","last_active":"2024-09-16T23:42:27.680721+0000","last_peered":"2024-09-16T23:42:27.680721+0000","last_clean":"2024-09-16T23:42:27.680721+0000","last_became_active":"2024-09-16T23:06:31.343661+0000","last_became_peered":"2024-09-16T23:06:31.343661+0000","last_unstale":"2024-09-16T23:42:27.680721+0000","last_undegraded":"2024-09-16T23:42:27.680721+0000","last_fullsized":"2024-09-16T23:42:27.680721+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:27.680649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:27.680649+0000","last_clean_scrub_stamp":"2024-09-16T23:42:27.680649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:23:52.171943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011018930000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"388'3","reported_seq":793,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:26.757424+0000","last_change":"2024-09-16T23:42:26.757424+0000","last_active":"2024-09-16T23:42:26.757424+0000","last_peered":"2024-09-16T23:42:26.757424+0000","last_clean":"2024-09-16T23:42:26.757424+0000","last_became_active":"2024-09-16T23:06:31.341230+0000","last_became_peered":"2024-09-16T23:06:31.341230+0000","last_unstale":"2024-09-16T23:42:26.757424+0000","last_undegraded":"2024-09-16T23:42:26.757424+0000","last_fullsized":"2024-09-16T23:42:26.757424+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'3","last_scrub_stamp":"2024-09-16T23:42:26.757345+0000","last_deep_scrub":"388'3","last_deep_scrub_stamp":"2024-09-16T23:42:26.757345+0000","last_clean_scrub_stamp":"2024-09-16T23:42:26.757345+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:26.864267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051118400000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":789,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:28.219594+0000","last_change":"2024-09-16T23:42:28.219594+0000","last_active":"2024-09-16T23:42:28.219594+0000","last_peered":"2024-09-16T23:42:28.219594+0000","last_clean":"2024-09-16T23:42:28.219594+0000","last_became_active":"2024-09-16T23:06:31.341529+0000","last_became_peered":"2024-09-16T23:06:31.341529+0000","last_unstale":"2024-09-16T23:42:28.219594+0000","last_undegraded":"2024-09-16T23:42:28.219594+0000","last_fullsized":"2024-09-16T23:42:28.219594+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:28.219512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:28.219512+0000","last_clean_scrub_stamp":"2024-09-16T23:42:28.219512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T07:28:23.631684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032923799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":790,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:27.170213+0000","last_change":"2024-09-16T23:42:27.170161+0000","last_active":"2024-09-16T23:42:27.170213+0000","last_peered":"2024-09-16T23:42:27.170213+0000","last_clean":"2024-09-16T23:42:27.170213+0000","last_became_active":"2024-09-16T23:06:31.341493+0000","last_became_peered":"2024-09-16T23:06:31.341493+0000","last_unstale":"2024-09-16T23:42:27.170213+0000","last_undegraded":"2024-09-16T23:42:27.170213+0000","last_fullsized":"2024-09-16T23:42:27.170213+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:27.170095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:27.170095+0000","last_clean_scrub_stamp":"2024-09-16T23:42:27.170095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T02:30:09.142439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034678600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"388'2","reported_seq":791,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:29.664755+0000","last_change":"2024-09-16T23:42:29.664755+0000","last_active":"2024-09-16T23:42:29.664755+0000","last_peered":"2024-09-16T23:42:29.664755+0000","last_clean":"2024-09-16T23:42:29.664755+0000","last_became_active":"2024-09-16T23:06:31.343379+0000","last_became_peered":"2024-09-16T23:06:31.343379+0000","last_unstale":"2024-09-16T23:42:29.664755+0000","last_undegraded":"2024-09-16T23:42:29.664755+0000","last_fullsized":"2024-09-16T23:42:29.664755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'2","last_scrub_stamp":"2024-09-16T23:42:29.664661+0000","last_deep_scrub":"388'2","last_deep_scrub_stamp":"2024-09-16T23:42:29.664661+0000","last_clean_scrub_stamp":"2024-09-16T23:42:29.664661+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:18:21.200677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00062375899999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"388'54","reported_seq":860,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-09-16T23:42:28.705741+0000","last_change":"2024-09-16T23:42:28.705657+0000","last_active":"2024-09-16T23:42:28.705741+0000","last_peered":"2024-09-16T23:42:28.705741+0000","last_clean":"2024-09-16T23:42:28.705741+0000","last_became_active":"2024-09-16T23:06:26.331015+0000","last_became_peered":"2024-09-16T23:06:26.331015+0000","last_unstale":"2024-09-16T23:42:28.705741+0000","last_undegraded":"2024-09-16T23:42:28.705741+0000","last_fullsized":"2024-09-16T23:42:28.705741+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'54","last_scrub_stamp":"2024-09-16T23:42:28.705533+0000","last_deep_scrub":"388'54","last_deep_scrub_stamp":"2024-09-16T23:42:28.705533+0000","last_clean_scrub_stamp":"2024-09-16T23:42:28.705533+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:21:55.357615+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11804,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738810,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1361280,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":1358142,"kb_avail":93010560,"statfs":{"total":96636764160,"available":95242813440,"internally_reserved":0,"allocated":3194880,"data_stored":1707017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1390737861},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1429436,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":1426238,"kb_avail":92942404,"statfs":{"total":96636764160,"available":95173021696,"internally_reserved":0,"allocated":3256320,"data_stored":2297404,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1460468165},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738809,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1642364,"kb_used_data":3180,"kb_used_omap":1,"kb_used_meta":1639166,"kb_avail":92729476,"statfs":{"total":96636764160,"available":94954983424,"internally_reserved":0,"allocated":3256320,"data_stored":2297404,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":1678506437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5902,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:47.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-16T23:42:48.020 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T23:42:48.021 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T23:42:48.021 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-16T23:42:48.021 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:48.081 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-16T23:42:48.081 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-16T23:42:48.081 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:48.151 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-16T23:42:48.151 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-16T23:42:48.151 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:48.216 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-16T23:42:48.216 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T23:42:48.217 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-16T23:42:48.217 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:48.247 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-16T23:42:48.247 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T23:42:48.247 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-16T23:42:48.248 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:42:48.260 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-16T23:42:48.260 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T23:42:48.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T23:42:48.380 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:42:48.380 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-16T23:42:48.721 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:42:48.721 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-16T23:42:48.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:42:48.850 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-16T23:42:48.977 INFO:tasks.ceph:Archiving mon data... 2024-09-16T23:42:48.979 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/data/cluster2.mon.a.tgz 2024-09-16T23:42:48.979 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-09-16T23:42:48.998 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.Z3dzyILtWR 2024-09-16T23:42:48.999 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.Z3dzyILtWR 2024-09-16T23:42:49.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.Z3dzyILtWR 2024-09-16T23:42:49.283 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.Z3dzyILtWR is 373KB 2024-09-16T23:42:49.379 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.Z3dzyILtWR 2024-09-16T23:42:49.395 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T23:42:49.395 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T23:42:49.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T23:42:49.458 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T23:42:49.469 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T23:42:49.469 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:42:49.757 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:42:49.758 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:42:49.770 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-09-16T23:42:49.273176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":243600,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":232251,"kb_avail":282871920,"statfs":{"total":289910292480,"available":289660846080,"internally_reserved":0,"allocated":11550720,"data_stored":7300978,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":237825359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002433"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640979+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:42:20.640979+0000","last_peered":"2024-09-16T23:42:20.640979+0000","last_clean":"2024-09-16T23:42:20.640979+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:42:20.640979+0000","last_undegraded":"2024-09-16T23:42:20.640979+0000","last_fullsized":"2024-09-16T23:42:20.640979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640032+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:42:20.640032+0000","last_peered":"2024-09-16T23:42:20.640032+0000","last_clean":"2024-09-16T23:42:20.640032+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:42:20.640032+0000","last_undegraded":"2024-09-16T23:42:20.640032+0000","last_fullsized":"2024-09-16T23:42:20.640032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642189+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:42:20.642189+0000","last_peered":"2024-09-16T23:42:20.642189+0000","last_clean":"2024-09-16T23:42:20.642189+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:42:20.642189+0000","last_undegraded":"2024-09-16T23:42:20.642189+0000","last_fullsized":"2024-09-16T23:42:20.642189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.637498+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:42:20.637498+0000","last_peered":"2024-09-16T23:42:20.637498+0000","last_clean":"2024-09-16T23:42:20.637498+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:42:20.637498+0000","last_undegraded":"2024-09-16T23:42:20.637498+0000","last_fullsized":"2024-09-16T23:42:20.637498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.805195+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:42:22.805195+0000","last_peered":"2024-09-16T23:42:22.805195+0000","last_clean":"2024-09-16T23:42:22.805195+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:42:22.805195+0000","last_undegraded":"2024-09-16T23:42:22.805195+0000","last_fullsized":"2024-09-16T23:42:22.805195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.803265+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:42:22.803265+0000","last_peered":"2024-09-16T23:42:22.803265+0000","last_clean":"2024-09-16T23:42:22.803265+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:42:22.803265+0000","last_undegraded":"2024-09-16T23:42:22.803265+0000","last_fullsized":"2024-09-16T23:42:22.803265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.806189+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:42:22.806189+0000","last_peered":"2024-09-16T23:42:22.806189+0000","last_clean":"2024-09-16T23:42:22.806189+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:42:22.806189+0000","last_undegraded":"2024-09-16T23:42:22.806189+0000","last_fullsized":"2024-09-16T23:42:22.806189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642470+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:42:20.642470+0000","last_peered":"2024-09-16T23:42:20.642470+0000","last_clean":"2024-09-16T23:42:20.642470+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:42:20.642470+0000","last_undegraded":"2024-09-16T23:42:20.642470+0000","last_fullsized":"2024-09-16T23:42:20.642470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":1002,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.641075+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:42:20.641075+0000","last_peered":"2024-09-16T23:42:20.641075+0000","last_clean":"2024-09-16T23:42:20.641075+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:42:20.641075+0000","last_undegraded":"2024-09-16T23:42:20.641075+0000","last_fullsized":"2024-09-16T23:42:20.641075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706113,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79388,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292452,"statfs":{"total":96636764160,"available":96555470848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79324,"kb_used_data":3720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292516,"statfs":{"total":96636764160,"available":96555536384,"internally_reserved":0,"allocated":3809280,"data_stored":2040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84888,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286952,"statfs":{"total":96636764160,"available":96549838848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:49.771 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:42:50.053 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:42:50.053 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:42:50.063 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-09-16T23:42:49.273176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":243600,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":232251,"kb_avail":282871920,"statfs":{"total":289910292480,"available":289660846080,"internally_reserved":0,"allocated":11550720,"data_stored":7300978,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":237825359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002433"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640979+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:42:20.640979+0000","last_peered":"2024-09-16T23:42:20.640979+0000","last_clean":"2024-09-16T23:42:20.640979+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:42:20.640979+0000","last_undegraded":"2024-09-16T23:42:20.640979+0000","last_fullsized":"2024-09-16T23:42:20.640979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640032+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:42:20.640032+0000","last_peered":"2024-09-16T23:42:20.640032+0000","last_clean":"2024-09-16T23:42:20.640032+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:42:20.640032+0000","last_undegraded":"2024-09-16T23:42:20.640032+0000","last_fullsized":"2024-09-16T23:42:20.640032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642189+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:42:20.642189+0000","last_peered":"2024-09-16T23:42:20.642189+0000","last_clean":"2024-09-16T23:42:20.642189+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:42:20.642189+0000","last_undegraded":"2024-09-16T23:42:20.642189+0000","last_fullsized":"2024-09-16T23:42:20.642189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.637498+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:42:20.637498+0000","last_peered":"2024-09-16T23:42:20.637498+0000","last_clean":"2024-09-16T23:42:20.637498+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:42:20.637498+0000","last_undegraded":"2024-09-16T23:42:20.637498+0000","last_fullsized":"2024-09-16T23:42:20.637498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.805195+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:42:22.805195+0000","last_peered":"2024-09-16T23:42:22.805195+0000","last_clean":"2024-09-16T23:42:22.805195+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:42:22.805195+0000","last_undegraded":"2024-09-16T23:42:22.805195+0000","last_fullsized":"2024-09-16T23:42:22.805195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.803265+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:42:22.803265+0000","last_peered":"2024-09-16T23:42:22.803265+0000","last_clean":"2024-09-16T23:42:22.803265+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:42:22.803265+0000","last_undegraded":"2024-09-16T23:42:22.803265+0000","last_fullsized":"2024-09-16T23:42:22.803265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.806189+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:42:22.806189+0000","last_peered":"2024-09-16T23:42:22.806189+0000","last_clean":"2024-09-16T23:42:22.806189+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:42:22.806189+0000","last_undegraded":"2024-09-16T23:42:22.806189+0000","last_fullsized":"2024-09-16T23:42:22.806189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642470+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:42:20.642470+0000","last_peered":"2024-09-16T23:42:20.642470+0000","last_clean":"2024-09-16T23:42:20.642470+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:42:20.642470+0000","last_undegraded":"2024-09-16T23:42:20.642470+0000","last_fullsized":"2024-09-16T23:42:20.642470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":1002,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.641075+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:42:20.641075+0000","last_peered":"2024-09-16T23:42:20.641075+0000","last_clean":"2024-09-16T23:42:20.641075+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:42:20.641075+0000","last_undegraded":"2024-09-16T23:42:20.641075+0000","last_fullsized":"2024-09-16T23:42:20.641075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706113,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79388,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292452,"statfs":{"total":96636764160,"available":96555470848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79324,"kb_used_data":3720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292516,"statfs":{"total":96636764160,"available":96555536384,"internally_reserved":0,"allocated":3809280,"data_stored":2040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84888,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286952,"statfs":{"total":96636764160,"available":96549838848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:50.064 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T23:42:50.065 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:42:50.343 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:42:50.344 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:42:50.356 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1573,"stamp":"2024-09-16T23:42:49.273176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":243600,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":232251,"kb_avail":282871920,"statfs":{"total":289910292480,"available":289660846080,"internally_reserved":0,"allocated":11550720,"data_stored":7300978,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":237825359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002433"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640979+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:42:20.640979+0000","last_peered":"2024-09-16T23:42:20.640979+0000","last_clean":"2024-09-16T23:42:20.640979+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:42:20.640979+0000","last_undegraded":"2024-09-16T23:42:20.640979+0000","last_fullsized":"2024-09-16T23:42:20.640979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640032+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:42:20.640032+0000","last_peered":"2024-09-16T23:42:20.640032+0000","last_clean":"2024-09-16T23:42:20.640032+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:42:20.640032+0000","last_undegraded":"2024-09-16T23:42:20.640032+0000","last_fullsized":"2024-09-16T23:42:20.640032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642189+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:42:20.642189+0000","last_peered":"2024-09-16T23:42:20.642189+0000","last_clean":"2024-09-16T23:42:20.642189+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:42:20.642189+0000","last_undegraded":"2024-09-16T23:42:20.642189+0000","last_fullsized":"2024-09-16T23:42:20.642189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.637498+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:42:20.637498+0000","last_peered":"2024-09-16T23:42:20.637498+0000","last_clean":"2024-09-16T23:42:20.637498+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:42:20.637498+0000","last_undegraded":"2024-09-16T23:42:20.637498+0000","last_fullsized":"2024-09-16T23:42:20.637498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.805195+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:42:22.805195+0000","last_peered":"2024-09-16T23:42:22.805195+0000","last_clean":"2024-09-16T23:42:22.805195+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:42:22.805195+0000","last_undegraded":"2024-09-16T23:42:22.805195+0000","last_fullsized":"2024-09-16T23:42:22.805195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.803265+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:42:22.803265+0000","last_peered":"2024-09-16T23:42:22.803265+0000","last_clean":"2024-09-16T23:42:22.803265+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:42:22.803265+0000","last_undegraded":"2024-09-16T23:42:22.803265+0000","last_fullsized":"2024-09-16T23:42:22.803265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.806189+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:42:22.806189+0000","last_peered":"2024-09-16T23:42:22.806189+0000","last_clean":"2024-09-16T23:42:22.806189+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:42:22.806189+0000","last_undegraded":"2024-09-16T23:42:22.806189+0000","last_fullsized":"2024-09-16T23:42:22.806189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642470+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:42:20.642470+0000","last_peered":"2024-09-16T23:42:20.642470+0000","last_clean":"2024-09-16T23:42:20.642470+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:42:20.642470+0000","last_undegraded":"2024-09-16T23:42:20.642470+0000","last_fullsized":"2024-09-16T23:42:20.642470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":1002,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.641075+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:42:20.641075+0000","last_peered":"2024-09-16T23:42:20.641075+0000","last_clean":"2024-09-16T23:42:20.641075+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:42:20.641075+0000","last_undegraded":"2024-09-16T23:42:20.641075+0000","last_fullsized":"2024-09-16T23:42:20.641075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706113,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79388,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292452,"statfs":{"total":96636764160,"available":96555470848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79324,"kb_used_data":3720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292516,"statfs":{"total":96636764160,"available":96555536384,"internally_reserved":0,"allocated":3809280,"data_stored":2040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84888,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286952,"statfs":{"total":96636764160,"available":96549838848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:50.357 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T23:42:50.641 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:42:50.641 INFO:teuthology.orchestra.run.smithi096.stdout:{"epoch":463,"fsid":"788e50d2-621a-4a2a-b433-20948746ee44","created":"2024-09-16T23:05:36.319966+0000","modified":"2024-09-16T23:42:20.631548+0000","last_up_change":"2024-09-16T23:05:45.707878+0000","last_in_change":"2024-09-16T23:05:37.166708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T23:05:44.711648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T23:05:51.483810+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"12a56727-19bf-4ba9-ba93-67a8c3f83bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6809","nonce":1838959797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6811","nonce":1838959797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6815","nonce":1838959797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":1838959797},{"type":"v1","addr":"172.21.15.96:6813","nonce":1838959797}]},"public_addr":"172.21.15.96:6809/1838959797","cluster_addr":"172.21.15.96:6811/1838959797","heartbeat_back_addr":"172.21.15.96:6815/1838959797","heartbeat_front_addr":"172.21.15.96:6813/1838959797","state":["exists","up"]},{"osd":1,"uuid":"cffe71c7-f3a6-474d-9b81-85f56cc55750","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6817","nonce":3258003119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6819","nonce":3258003119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6823","nonce":3258003119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":3258003119},{"type":"v1","addr":"172.21.15.96:6821","nonce":3258003119}]},"public_addr":"172.21.15.96:6817/3258003119","cluster_addr":"172.21.15.96:6819/3258003119","heartbeat_back_addr":"172.21.15.96:6823/3258003119","heartbeat_front_addr":"172.21.15.96:6821/3258003119","state":["exists","up"]},{"osd":2,"uuid":"5d99f04e-e5b2-4b72-b9d4-3d29868fbb40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6801","nonce":1539697306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6803","nonce":1539697306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6807","nonce":1539697306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":1539697306},{"type":"v1","addr":"172.21.15.96:6805","nonce":1539697306}]},"public_addr":"172.21.15.96:6801/1539697306","cluster_addr":"172.21.15.96:6803/1539697306","heartbeat_back_addr":"172.21.15.96:6807/1539697306","heartbeat_front_addr":"172.21.15.96:6805/1539697306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.473113+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.729249+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T23:05:43.626380+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T23:42:51.654 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-16T23:42:51.655 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:51.789 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-16T23:42:51.790 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:51.790 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-16T23:42:51.800 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-16T23:42:52.084 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T23:42:52.100 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-16T23:42:52.100 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:52.222 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-16T23:42:52.222 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:52.222 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-16T23:42:52.232 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-16T23:42:52.514 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T23:42:52.527 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-16T23:42:52.527 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T23:42:52.648 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-16T23:42:52.648 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T23:42:52.649 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-16T23:42:52.659 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-16T23:42:52.941 INFO:teuthology.orchestra.run.smithi096.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T23:42:52.954 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:42:53.233 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:42:53.233 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:42:53.245 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2024-09-16T23:42:51.273543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":243600,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":232251,"kb_avail":282871920,"statfs":{"total":289910292480,"available":289660846080,"internally_reserved":0,"allocated":11550720,"data_stored":7300978,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":237825359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002323"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640979+0000","last_change":"2024-09-16T23:05:54.772641+0000","last_active":"2024-09-16T23:42:20.640979+0000","last_peered":"2024-09-16T23:42:20.640979+0000","last_clean":"2024-09-16T23:42:20.640979+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:42:20.640979+0000","last_undegraded":"2024-09-16T23:42:20.640979+0000","last_fullsized":"2024-09-16T23:42:20.640979+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:07:57.222895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.640032+0000","last_change":"2024-09-16T23:05:54.772355+0000","last_active":"2024-09-16T23:42:20.640032+0000","last_peered":"2024-09-16T23:42:20.640032+0000","last_clean":"2024-09-16T23:42:20.640032+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:42:20.640032+0000","last_undegraded":"2024-09-16T23:42:20.640032+0000","last_fullsized":"2024-09-16T23:42:20.640032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:41:28.158116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642189+0000","last_change":"2024-09-16T23:05:54.772387+0000","last_active":"2024-09-16T23:42:20.642189+0000","last_peered":"2024-09-16T23:42:20.642189+0000","last_clean":"2024-09-16T23:42:20.642189+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:42:20.642189+0000","last_undegraded":"2024-09-16T23:42:20.642189+0000","last_fullsized":"2024-09-16T23:42:20.642189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T03:44:44.814444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.637498+0000","last_change":"2024-09-16T23:05:54.772071+0000","last_active":"2024-09-16T23:42:20.637498+0000","last_peered":"2024-09-16T23:42:20.637498+0000","last_clean":"2024-09-16T23:42:20.637498+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:42:20.637498+0000","last_undegraded":"2024-09-16T23:42:20.637498+0000","last_fullsized":"2024-09-16T23:42:20.637498+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:06:37.943550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.805195+0000","last_change":"2024-09-16T23:05:54.769089+0000","last_active":"2024-09-16T23:42:22.805195+0000","last_peered":"2024-09-16T23:42:22.805195+0000","last_clean":"2024-09-16T23:42:22.805195+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:42:22.805195+0000","last_undegraded":"2024-09-16T23:42:22.805195+0000","last_fullsized":"2024-09-16T23:42:22.805195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:31:13.255064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":933,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.803265+0000","last_change":"2024-09-16T23:05:54.765826+0000","last_active":"2024-09-16T23:42:22.803265+0000","last_peered":"2024-09-16T23:42:22.803265+0000","last_clean":"2024-09-16T23:42:22.803265+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:42:22.803265+0000","last_undegraded":"2024-09-16T23:42:22.803265+0000","last_fullsized":"2024-09-16T23:42:22.803265+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:44:08.362529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:22.806189+0000","last_change":"2024-09-16T23:05:54.765915+0000","last_active":"2024-09-16T23:42:22.806189+0000","last_peered":"2024-09-16T23:42:22.806189+0000","last_clean":"2024-09-16T23:42:22.806189+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:42:22.806189+0000","last_undegraded":"2024-09-16T23:42:22.806189+0000","last_fullsized":"2024-09-16T23:42:22.806189+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:16:36.603674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":934,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.642470+0000","last_change":"2024-09-16T23:05:54.772010+0000","last_active":"2024-09-16T23:42:20.642470+0000","last_peered":"2024-09-16T23:42:20.642470+0000","last_clean":"2024-09-16T23:42:20.642470+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:42:20.642470+0000","last_undegraded":"2024-09-16T23:42:20.642470+0000","last_fullsized":"2024-09-16T23:42:20.642470+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:51.752330+0000","last_clean_scrub_stamp":"2024-09-16T23:05:51.752330+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:22:58.552116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'52","reported_seq":1002,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:20.641075+0000","last_change":"2024-09-16T23:05:47.481519+0000","last_active":"2024-09-16T23:42:20.641075+0000","last_peered":"2024-09-16T23:42:20.641075+0000","last_clean":"2024-09-16T23:42:20.641075+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:42:20.641075+0000","last_undegraded":"2024-09-16T23:42:20.641075+0000","last_fullsized":"2024-09-16T23:42:20.641075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:05:45.707878+0000","last_clean_scrub_stamp":"2024-09-16T23:05:45.707878+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T06:18:21.058517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706113,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79388,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292452,"statfs":{"total":96636764160,"available":96555470848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79324,"kb_used_data":3720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292516,"statfs":{"total":96636764160,"available":96555536384,"internally_reserved":0,"allocated":3809280,"data_stored":2040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84888,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286952,"statfs":{"total":96636764160,"available":96549838848,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:42:53.246 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.246 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.246 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T23:05:51.752330+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=51, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T23:05:45.707878+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=5, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=23, tm_min=42, tm_sec=50, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T23:42:53.247 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T23:43:13.249 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T23:43:13.544 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-16T23:43:13.545 INFO:teuthology.orchestra.run.smithi096.stderr:dumped all 2024-09-16T23:43:13.556 INFO:teuthology.orchestra.run.smithi096.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2024-09-16T23:43:13.277600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":243584,"kb_used_data":11280,"kb_used_omap":4,"kb_used_meta":232251,"kb_avail":282871936,"statfs":{"total":289910292480,"available":289660862464,"internally_reserved":0,"allocated":11550720,"data_stored":7300978,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":4785,"internal_metadata":237825359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002204"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:55.948373+0000","last_change":"2024-09-16T23:42:55.948373+0000","last_active":"2024-09-16T23:42:55.948373+0000","last_peered":"2024-09-16T23:42:55.948373+0000","last_clean":"2024-09-16T23:42:55.948373+0000","last_became_active":"2024-09-16T23:05:52.768454+0000","last_became_peered":"2024-09-16T23:05:52.768454+0000","last_unstale":"2024-09-16T23:42:55.948373+0000","last_undegraded":"2024-09-16T23:42:55.948373+0000","last_fullsized":"2024-09-16T23:42:55.948373+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:55.948288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:55.948288+0000","last_clean_scrub_stamp":"2024-09-16T23:42:55.948288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:55:27.440761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001099746,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:56.954967+0000","last_change":"2024-09-16T23:42:56.954967+0000","last_active":"2024-09-16T23:42:56.954967+0000","last_peered":"2024-09-16T23:42:56.954967+0000","last_clean":"2024-09-16T23:42:56.954967+0000","last_became_active":"2024-09-16T23:05:52.768382+0000","last_became_peered":"2024-09-16T23:05:52.768382+0000","last_unstale":"2024-09-16T23:42:56.954967+0000","last_undegraded":"2024-09-16T23:42:56.954967+0000","last_fullsized":"2024-09-16T23:42:56.954967+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:56.954877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:56.954877+0000","last_clean_scrub_stamp":"2024-09-16T23:42:56.954877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:11:04.249199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087633600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:53.869458+0000","last_change":"2024-09-16T23:42:53.869458+0000","last_active":"2024-09-16T23:42:53.869458+0000","last_peered":"2024-09-16T23:42:53.869458+0000","last_clean":"2024-09-16T23:42:53.869458+0000","last_became_active":"2024-09-16T23:05:52.768939+0000","last_became_peered":"2024-09-16T23:05:52.768939+0000","last_unstale":"2024-09-16T23:42:53.869458+0000","last_undegraded":"2024-09-16T23:42:53.869458+0000","last_fullsized":"2024-09-16T23:42:53.869458+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:53.869322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:53.869322+0000","last_clean_scrub_stamp":"2024-09-16T23:42:53.869322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:07:50.597166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099375799999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:54.919366+0000","last_change":"2024-09-16T23:42:54.919366+0000","last_active":"2024-09-16T23:42:54.919366+0000","last_peered":"2024-09-16T23:42:54.919366+0000","last_clean":"2024-09-16T23:42:54.919366+0000","last_became_active":"2024-09-16T23:05:52.768738+0000","last_became_peered":"2024-09-16T23:05:52.768738+0000","last_unstale":"2024-09-16T23:42:54.919366+0000","last_undegraded":"2024-09-16T23:42:54.919366+0000","last_fullsized":"2024-09-16T23:42:54.919366+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:54.919276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:54.919276+0000","last_clean_scrub_stamp":"2024-09-16T23:42:54.919276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:07:28.287608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067743399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"463'3","reported_seq":942,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:52.721583+0000","last_change":"2024-09-16T23:42:52.721583+0000","last_active":"2024-09-16T23:42:52.721583+0000","last_peered":"2024-09-16T23:42:52.721583+0000","last_clean":"2024-09-16T23:42:52.721583+0000","last_became_active":"2024-09-16T23:05:52.764463+0000","last_became_peered":"2024-09-16T23:05:52.764463+0000","last_unstale":"2024-09-16T23:42:52.721583+0000","last_undegraded":"2024-09-16T23:42:52.721583+0000","last_fullsized":"2024-09-16T23:42:52.721583+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'3","last_scrub_stamp":"2024-09-16T23:42:52.721520+0000","last_deep_scrub":"463'3","last_deep_scrub_stamp":"2024-09-16T23:42:52.721520+0000","last_clean_scrub_stamp":"2024-09-16T23:42:52.721520+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T09:34:45.847057+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001072029,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":941,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:54.846465+0000","last_change":"2024-09-16T23:42:54.846465+0000","last_active":"2024-09-16T23:42:54.846465+0000","last_peered":"2024-09-16T23:42:54.846465+0000","last_clean":"2024-09-16T23:42:54.846465+0000","last_became_active":"2024-09-16T23:05:52.764473+0000","last_became_peered":"2024-09-16T23:05:52.764473+0000","last_unstale":"2024-09-16T23:42:54.846465+0000","last_undegraded":"2024-09-16T23:42:54.846465+0000","last_fullsized":"2024-09-16T23:42:54.846465+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:54.846354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:54.846354+0000","last_clean_scrub_stamp":"2024-09-16T23:42:54.846354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T05:30:09.319456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049196299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":942,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:53.866388+0000","last_change":"2024-09-16T23:42:53.866388+0000","last_active":"2024-09-16T23:42:53.866388+0000","last_peered":"2024-09-16T23:42:53.866388+0000","last_clean":"2024-09-16T23:42:53.866388+0000","last_became_active":"2024-09-16T23:05:52.764544+0000","last_became_peered":"2024-09-16T23:05:52.764544+0000","last_unstale":"2024-09-16T23:42:53.866388+0000","last_undegraded":"2024-09-16T23:42:53.866388+0000","last_fullsized":"2024-09-16T23:42:53.866388+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T23:42:53.866301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T23:42:53.866301+0000","last_clean_scrub_stamp":"2024-09-16T23:42:53.866301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T11:12:16.142452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000571303,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"463'2","reported_seq":943,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:52.905269+0000","last_change":"2024-09-16T23:42:52.905269+0000","last_active":"2024-09-16T23:42:52.905269+0000","last_peered":"2024-09-16T23:42:52.905269+0000","last_clean":"2024-09-16T23:42:52.905269+0000","last_became_active":"2024-09-16T23:05:52.768833+0000","last_became_peered":"2024-09-16T23:05:52.768833+0000","last_unstale":"2024-09-16T23:42:52.905269+0000","last_undegraded":"2024-09-16T23:42:52.905269+0000","last_fullsized":"2024-09-16T23:42:52.905269+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'2","last_scrub_stamp":"2024-09-16T23:42:52.905197+0000","last_deep_scrub":"463'2","last_deep_scrub_stamp":"2024-09-16T23:42:52.905197+0000","last_clean_scrub_stamp":"2024-09-16T23:42:52.905197+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T10:12:10.137009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00063210499999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"463'54","reported_seq":1012,"reported_epoch":463,"state":"active+clean","last_fresh":"2024-09-16T23:42:57.986131+0000","last_change":"2024-09-16T23:42:57.986131+0000","last_active":"2024-09-16T23:42:57.986131+0000","last_peered":"2024-09-16T23:42:57.986131+0000","last_clean":"2024-09-16T23:42:57.986131+0000","last_became_active":"2024-09-16T23:05:47.480598+0000","last_became_peered":"2024-09-16T23:05:47.480598+0000","last_unstale":"2024-09-16T23:42:57.986131+0000","last_undegraded":"2024-09-16T23:42:57.986131+0000","last_fullsized":"2024-09-16T23:42:57.986131+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'54","last_scrub_stamp":"2024-09-16T23:42:57.986035+0000","last_deep_scrub":"463'54","last_deep_scrub_stamp":"2024-09-16T23:42:57.986035+0000","last_clean_scrub_stamp":"2024-09-16T23:42:57.986035+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T04:12:40.152625+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":11796,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654706118,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79380,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292460,"statfs":{"total":96636764160,"available":96555479040,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79320,"kb_used_data":3720,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292520,"statfs":{"total":96636764160,"available":96555540480,"internally_reserved":0,"allocated":3809280,"data_stored":2040068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84884,"kb_used_data":3780,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94286956,"statfs":{"total":96636764160,"available":96549842944,"internally_reserved":0,"allocated":3870720,"data_stored":2630455,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":5898,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T23:43:13.558 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-16T23:43:13.861 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T23:43:13.861 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T23:43:13.862 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-16T23:43:13.862 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:43:13.905 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-16T23:43:13.905 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-16T23:43:13.906 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:43:13.941 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-16T23:43:13.942 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-16T23:43:13.942 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:43:14.022 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-16T23:43:14.023 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T23:43:14.023 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-16T23:43:14.023 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:43:14.089 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-16T23:43:14.089 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T23:43:14.089 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-16T23:43:14.089 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T23:43:14.142 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-16T23:43:14.142 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T23:43:14.142 DEBUG:teuthology.orchestra.run.smithi096:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T23:43:14.210 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:43:14.210 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-16T23:43:14.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:43:14.389 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-16T23:43:14.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:43:14.500 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-16T23:43:14.566 INFO:tasks.ceph:Archiving mon data... 2024-09-16T23:43:14.566 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/data/cluster1.mon.a.tgz 2024-09-16T23:43:14.567 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-16T23:43:14.582 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.6A0cTxmtxd 2024-09-16T23:43:14.582 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.6A0cTxmtxd 2024-09-16T23:43:14.737 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.6A0cTxmtxd 2024-09-16T23:43:14.820 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.6A0cTxmtxd is 410KB 2024-09-16T23:43:14.889 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.6A0cTxmtxd 2024-09-16T23:43:14.904 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T23:43:14.904 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T23:43:14.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T23:43:15.023 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T23:43:15.042 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T23:43:15.042 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T23:43:15.044 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi096/crash 2024-09-16T23:43:15.045 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T23:43:15.088 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi098/crash 2024-09-16T23:43:15.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T23:43:15.118 INFO:tasks.ceph:Compressing logs... 2024-09-16T23:43:15.118 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T23:43:15.133 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37626.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35336.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37626.log.gz 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35888.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35336.log.gz 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37502.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35510.log 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35888.log.gz 2024-09-16T23:43:15.167 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37502.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36098.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35510.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37460.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36098.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.36098.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36014.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37334.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37460.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36014.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster1-client.admin.36014.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/cluster1-client.admin.32255.log 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37334.log: gzip -5 --verbose -- 0.0% /var/log/ceph/cluster1-client.admin.35675.log -- replaced with /var/log/ceph/cluster1-client.admin.37334.log.gz 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-16T23:43:15.168 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32057.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32255.log.gz 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35675.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35675.log.gz -5 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37586.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35762.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32057.log.gz 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35908.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37586.log.gz 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35762.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35762.log.gz 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36056.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35908.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35908.log.gzgzip 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.35948.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.36056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36056.log.gz 2024-09-16T23:43:15.169 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35948.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32295.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35948.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35802.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32295.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32295.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35552.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35802.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35802.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37124.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32215.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35552.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37166.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37124.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32215.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27565.log 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32215.log.gz 2024-09-16T23:43:15.170 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37166.log.gz 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.tmp-client.admin.27565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35674.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27565.log.gz 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32394.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35674.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35674.log.gz 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr: /var/log/ceph/cluster1-client.admin.35075.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32394.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32394.log.gz 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35075.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35075.log.gz 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.37208.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37208.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32459.log 2024-09-16T23:43:15.171 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37208.log.gz 2024-09-16T23:43:15.172 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-osd.0.log: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-16T23:43:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35801.log 2024-09-16T23:43:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.32459.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35392.log 2024-09-16T23:43:15.180 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.32459.log.gz 2024-09-16T23:43:15.180 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35801.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35801.log.gz 2024-09-16T23:43:15.186 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35468.log 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35392.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35392.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37250.log 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37292.log 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35468.log: /var/log/ceph/cluster1-client.admin.37250.log: 0.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35468.log.gz 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.37250.log.gz 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37418.log 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35676.log 2024-09-16T23:43:15.188 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37418.log: /var/log/ceph/cluster1-client.admin.37292.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37292.log.gz 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37418.log.gz 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35594.log 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35594.log: /var/log/ceph/cluster1-client.admin.35676.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35676.log.gz 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.35594.log.gz 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35417.log 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37376.log 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.35417.log: /var/log/ceph/cluster1.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.35417.log.gz 2024-09-16T23:43:15.189 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37544.log 2024-09-16T23:43:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37376.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37376.log.gz 2024-09-16T23:43:15.190 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/cluster1-client.admin.37544.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37544.log.gz 2024-09-16T23:43:15.190 INFO:teuthology.orchestra.run.smithi096.stderr: 87.8% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-16T23:43:15.204 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70166.log 2024-09-16T23:43:15.204 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68840.log 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41843.log 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70166.log.gz 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68007.log 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68840.log.gz 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58944.log 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68007.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65262.log 2024-09-16T23:43:15.208 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68007.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58944.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61284.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58944.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42019.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65262.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61284.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65530.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61172.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61284.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65530.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44431.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61172.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56871.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61172.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72815.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69899.log 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65530.log.gz 2024-09-16T23:43:15.209 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68208.log 2024-09-16T23:43:15.218 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56871.log.gz 2024-09-16T23:43:15.218 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72815.log.gz 2024-09-16T23:43:15.218 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41932.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69899.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55625.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68208.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36530.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71859.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55625.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.36530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60875.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36530.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54251.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46064.log 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71859.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60875.log.gz 2024-09-16T23:43:15.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54359.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65675.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54359.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56381.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46064.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54359.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.39491.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54251.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56381.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65675.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67135.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.2.39491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49390.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56381.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51654.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67135.log.gz 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43806.log 2024-09-16T23:43:15.220 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59931.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63931.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51654.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40158.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71169.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59931.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63931.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49539.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40158.log: /var/log/ceph/cluster2-client.mirror.71169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47572.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49539.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71169.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32902.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47572.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41843.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32902.log: gzip -5 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32902.log.gz 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.70485.log 2024-09-16T23:43:15.221 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49390.log.gz 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr: 79.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46024.log 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49539.log.gz 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43481.log 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70485.log.gz 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73920.log 2024-09-16T23:43:15.222 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46024.log: /var/log/ceph/cluster1-client.mirror.43481.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43481.log.gz 2024-09-16T23:43:15.236 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44431.log.gz 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41932.log.gz 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57361.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4%/var/log/ceph/cluster1-client.mirror.73920.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43806.log.gz 0.0% 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.73920.log.gz 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39848.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/cluster2-client.mirror.57361.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45902.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44451.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57361.log.gz 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43055.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44451.log: gzip 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45902.log.gz 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70351.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73320.log 2024-09-16T23:43:15.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70351.log: gzip -5 82.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70351.log.gz 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.62391.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69813.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73320.log.gz/var/log/ceph/cluster2-client.mirror.62391.log: 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73217.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59439.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.69813.log.gz 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64186.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62391.log.gz 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73217.log.gz 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.59439.log: --verbose -- /var/log/ceph/cluster2-client.mirror.58270.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44803.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64186.log.gz 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59909.log 2024-09-16T23:43:15.238 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44451.log.gz 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58270.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59439.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44803.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43055.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74248.log 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59909.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44803.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45153.log 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74248.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74248.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46591.log 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.46143.log 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.68009.log 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46591.log.gz/var/log/ceph/cluster2-client.mirror.3.46143.log: 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59909.log.gz 2024-09-16T23:43:15.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69364.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.1.68009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37984.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68717.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69364.log.gz 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.37984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62322.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60971.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68717.log.gz 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62322.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62322.log.gz 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71269.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59796.log 2024-09-16T23:43:15.240 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60971.log.gz 2024-09-16T23:43:15.250 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71269.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.68009.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63811.log 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59796.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59796.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50674.log 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40523.log 62.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37984.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50674.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.39491.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50674.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63811.log.gz 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61872.log 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.40523.log: -- /var/log/ceph/cluster2-client.mirror.47717.log 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53158.log 2024-09-16T23:43:15.251 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47717.log: /var/log/ceph/cluster1-client.mirror.61872.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61872.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47717.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40158.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61824.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68881.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61824.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53158.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61824.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61533.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68696.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68881.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74458.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61533.log.gz/var/log/ceph/cluster2-client.mirror.68696.log: 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68696.log.gz 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42871.log 2024-09-16T23:43:15.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44279.log 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74458.log.gz 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57208.log 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.42871.log.gz 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50715.log 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39848.log.gz 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57208.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57208.log.gz 2024-09-16T23:43:15.253 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70690.log 2024-09-16T23:43:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50715.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50715.log.gz 2024-09-16T23:43:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54580.log 2024-09-16T23:43:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70690.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70690.log.gz 2024-09-16T23:43:15.260 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44279.log.gz 2024-09-16T23:43:15.260 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58920.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61647.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54580.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39870.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61647.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40523.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61647.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72649.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73135.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72649.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58920.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.72649.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44863.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46883.log 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44863.log: /var/log/ceph/cluster2-client.mirror.73135.log: 87.8% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44863.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73135.log.gz 2024-09-16T23:43:15.261 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44603.log 2024-09-16T23:43:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59752.log 2024-09-16T23:43:15.262 INFO:teuthology.orchestra.run.smithi098.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46883.log.gz 2024-09-16T23:43:15.262 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44603.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44603.log.gz 2024-09-16T23:43:15.268 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51630.log 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.59752.log: -- /var/log/ceph/cluster2-client.mirror.45467.log 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51630.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59752.log.gz 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51630.log.gz 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66505.log 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72264.log 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45467.log.gz 2024-09-16T23:43:15.269 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66505.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66505.log.gz 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53825.log 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51851.log 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53825.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72264.log.gz 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53825.log.gz 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40571.log 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72584.log 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50405.log 2024-09-16T23:43:15.270 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40571.log.gz 2024-09-16T23:43:15.271 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72584.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72584.log.gz 2024-09-16T23:43:15.277 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56646.log 2024-09-16T23:43:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50405.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50405.log.gz 2024-09-16T23:43:15.278 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59844.log 2024-09-16T23:43:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56646.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56646.log.gz 2024-09-16T23:43:15.280 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40904.log 2024-09-16T23:43:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59844.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59844.log.gz 2024-09-16T23:43:15.281 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40930.log 2024-09-16T23:43:15.282 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.40904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51185.log 2024-09-16T23:43:15.283 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42561.log 2024-09-16T23:43:15.284 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51185.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40904.log.gz 2024-09-16T23:43:15.284 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.40930.log.gz 2024-09-16T23:43:15.286 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51185.log.gz 2024-09-16T23:43:15.290 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51851.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51851.log.gz 2024-09-16T23:43:15.292 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54407.log 2024-09-16T23:43:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42561.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42561.log.gz 2024-09-16T23:43:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64331.log 2024-09-16T23:43:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54407.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54407.log.gz 2024-09-16T23:43:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43075.log 2024-09-16T23:43:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64331.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64331.log.gz 2024-09-16T23:43:15.301 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46373.log 2024-09-16T23:43:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43075.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43075.log.gz 2024-09-16T23:43:15.307 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67832.log 2024-09-16T23:43:15.321 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64517.log 2024-09-16T23:43:15.322 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67832.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67832.log.gz 2024-09-16T23:43:15.328 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73575.log 2024-09-16T23:43:15.331 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64517.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.46143.log.gz 2024-09-16T23:43:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56542.log 2024-09-16T23:43:15.332 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73575.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73575.log.gz 2024-09-16T23:43:15.333 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64517.log.gz 2024-09-16T23:43:15.333 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40660.log 2024-09-16T23:43:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56542.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56542.log.gz 2024-09-16T23:43:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36150.log 2024-09-16T23:43:15.336 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48502.log 2024-09-16T23:43:15.336 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36150.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36150.log.gz 2024-09-16T23:43:15.337 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36978.log 2024-09-16T23:43:15.346 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68819.log 2024-09-16T23:43:15.349 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.36978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51269.log 2024-09-16T23:43:15.349 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68819.log: 16.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36978.log.gz 2024-09-16T23:43:15.349 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68819.log.gz 2024-09-16T23:43:15.349 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73299.log 2024-09-16T23:43:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51269.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51269.log.gz 2024-09-16T23:43:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51582.log 2024-09-16T23:43:15.351 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73299.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73299.log.gz 2024-09-16T23:43:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59346.log 2024-09-16T23:43:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51582.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51582.log.gz 2024-09-16T23:43:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61034.log 2024-09-16T23:43:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59346.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59346.log.gz 2024-09-16T23:43:15.356 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56694.log 2024-09-16T23:43:15.357 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61034.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61034.log.gz 2024-09-16T23:43:15.357 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51534.log 2024-09-16T23:43:15.359 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56694.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56694.log.gz 2024-09-16T23:43:15.359 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63596.log 2024-09-16T23:43:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51534.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51534.log.gz 2024-09-16T23:43:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68425.log 2024-09-16T23:43:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63596.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63596.log.gz 2024-09-16T23:43:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47104.log 2024-09-16T23:43:15.361 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68425.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68425.log.gz 2024-09-16T23:43:15.363 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52072.log 2024-09-16T23:43:15.363 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47104.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47104.log.gz 2024-09-16T23:43:15.363 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69007.log 2024-09-16T23:43:15.364 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52072.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52072.log.gz 2024-09-16T23:43:15.366 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44783.log 2024-09-16T23:43:15.367 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69007.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69007.log.gz 2024-09-16T23:43:15.367 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71937.log 2024-09-16T23:43:15.368 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44783.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44783.log.gz 2024-09-16T23:43:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71066.log 2024-09-16T23:43:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71937.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71937.log.gz 2024-09-16T23:43:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65054.log 2024-09-16T23:43:15.370 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71066.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71066.log.gz 2024-09-16T23:43:15.371 INFO:teuthology.orchestra.run.smithi096.stderr: 91.1% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-16T23:43:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61983.log 2024-09-16T23:43:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65054.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65054.log.gz 2024-09-16T23:43:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64062.log 2024-09-16T23:43:15.373 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61983.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61983.log.gz 2024-09-16T23:43:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61491.log 2024-09-16T23:43:15.374 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64062.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64062.log.gz 2024-09-16T23:43:15.375 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57477.log 2024-09-16T23:43:15.377 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61491.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61491.log.gz 2024-09-16T23:43:15.377 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50429.log 2024-09-16T23:43:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57477.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57477.log.gz 2024-09-16T23:43:15.378 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41176.log 2024-09-16T23:43:15.379 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50429.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50429.log.gz 2024-09-16T23:43:15.380 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39363.log 2024-09-16T23:43:15.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41176.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64951.log 2024-09-16T23:43:15.381 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53741.log 2024-09-16T23:43:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64951.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64951.log.gz 2024-09-16T23:43:15.382 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40660.log.gz 2024-09-16T23:43:15.382 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51920.log 2024-09-16T23:43:15.384 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53741.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53741.log.gz 2024-09-16T23:43:15.385 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43565.log 2024-09-16T23:43:15.386 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51920.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51920.log.gz 2024-09-16T23:43:15.387 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53090.log 2024-09-16T23:43:15.393 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43565.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43565.log.gz 2024-09-16T23:43:15.393 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64248.log 2024-09-16T23:43:15.394 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53090.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53090.log.gz 2024-09-16T23:43:15.394 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62571.log 2024-09-16T23:43:15.395 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64248.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64248.log.gz 2024-09-16T23:43:15.396 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52675.log 2024-09-16T23:43:15.396 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62571.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.62571.log.gz 2024-09-16T23:43:15.397 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66865.log 2024-09-16T23:43:15.399 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52675.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52675.log.gz 2024-09-16T23:43:15.400 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42049.log 2024-09-16T23:43:15.401 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66865.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66865.log.gz 2024-09-16T23:43:15.401 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70314.log 2024-09-16T23:43:15.401 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.42049.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42049.log.gz 2024-09-16T23:43:15.402 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58784.log 2024-09-16T23:43:15.403 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70314.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70314.log.gz 2024-09-16T23:43:15.404 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59684.log 2024-09-16T23:43:15.406 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58784.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58784.log.gz 2024-09-16T23:43:15.406 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45173.log 2024-09-16T23:43:15.409 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59684.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59684.log.gz 2024-09-16T23:43:15.409 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72773.log 2024-09-16T23:43:15.410 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45173.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45173.log.gz 2024-09-16T23:43:15.410 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55935.log 2024-09-16T23:43:15.411 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72773.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72773.log.gz 2024-09-16T23:43:15.411 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32839.log 2024-09-16T23:43:15.412 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55935.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55935.log.gz 2024-09-16T23:43:15.412 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59418.log 2024-09-16T23:43:15.414 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32839.log.gz 2024-09-16T23:43:15.414 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50919.log 2024-09-16T23:43:15.414 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59418.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59418.log.gz 2024-09-16T23:43:15.415 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50960.log 2024-09-16T23:43:15.416 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50919.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50919.log.gz 2024-09-16T23:43:15.416 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70063.log 2024-09-16T23:43:15.419 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50960.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50960.log.gz 2024-09-16T23:43:15.419 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55142.log 2024-09-16T23:43:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70063.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70063.log.gz 2024-09-16T23:43:15.420 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62274.log 2024-09-16T23:43:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55142.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55142.log.gz 2024-09-16T23:43:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49848.log 2024-09-16T23:43:15.422 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62274.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62274.log.gz 2024-09-16T23:43:15.423 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60264.log 2024-09-16T23:43:15.433 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49848.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42740.log 2024-09-16T23:43:15.440 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60264.log: 92.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69134.log 2024-09-16T23:43:15.440 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60264.log.gz 2024-09-16T23:43:15.452 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.2.42740.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41176.log.gz 2024-09-16T23:43:15.453 INFO:teuthology.orchestra.run.smithi096.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-16T23:43:15.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69384.log 2024-09-16T23:43:15.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69134.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69134.log.gz 2024-09-16T23:43:15.458 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42740.log.gz 2024-09-16T23:43:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44408.log 2024-09-16T23:43:15.470 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69384.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69384.log.gz 2024-09-16T23:43:15.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38125.log 2024-09-16T23:43:15.482 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44408.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44408.log.gz 2024-09-16T23:43:15.489 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70084.log 2024-09-16T23:43:15.496 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54162.log 2024-09-16T23:43:15.498 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70084.log: 87.7% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38125.log.gz 2024-09-16T23:43:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62849.log 2024-09-16T23:43:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54162.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70084.log.gz 2024-09-16T23:43:15.516 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45406.log 2024-09-16T23:43:15.517 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62849.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54162.log.gz 2024-09-16T23:43:15.518 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62849.log.gz 2024-09-16T23:43:15.524 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42538.log 2024-09-16T23:43:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45406.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43032.log 2024-09-16T23:43:15.537 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42538.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45406.log.gz 2024-09-16T23:43:15.542 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47433.log 2024-09-16T23:43:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43032.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42538.log.gz 2024-09-16T23:43:15.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48559.log 2024-09-16T23:43:15.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47433.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47572.log.gz 2024-09-16T23:43:15.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44302.log 2024-09-16T23:43:15.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48559.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48559.log.gz 2024-09-16T23:43:15.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51875.log 2024-09-16T23:43:15.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44302.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43032.log.gz 2024-09-16T23:43:15.561 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44302.log.gz 2024-09-16T23:43:15.562 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65448.log 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51875.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51875.log.gz 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49214.log 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65448.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47433.log.gz 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65448.log.gz 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68737.log 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49214.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49214.log.gz 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.70351.log 2024-09-16T23:43:15.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68737.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68737.log.gz 2024-09-16T23:43:15.567 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47636.log 2024-09-16T23:43:15.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.70351.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.70351.log.gz 2024-09-16T23:43:15.569 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50271.log 2024-09-16T23:43:15.571 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47636.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47636.log.gz 2024-09-16T23:43:15.571 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63724.log 2024-09-16T23:43:15.571 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50271.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50271.log.gz 2024-09-16T23:43:15.574 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50227.log 2024-09-16T23:43:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63724.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63724.log.gz 2024-09-16T23:43:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60244.log 2024-09-16T23:43:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50227.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50227.log.gz 2024-09-16T23:43:15.575 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64351.log 2024-09-16T23:43:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60244.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60244.log.gz 2024-09-16T23:43:15.577 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60424.log 2024-09-16T23:43:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64351.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64351.log.gz 2024-09-16T23:43:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43522.log 2024-09-16T23:43:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60424.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60424.log.gz 2024-09-16T23:43:15.580 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68940.log 2024-09-16T23:43:15.586 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43522.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43522.log.gz 2024-09-16T23:43:15.586 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63368.log 2024-09-16T23:43:15.601 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68940.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42019.log.gz 2024-09-16T23:43:15.602 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48908.log 2024-09-16T23:43:15.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63368.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45153.log.gz 2024-09-16T23:43:15.608 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63368.log.gz 2024-09-16T23:43:15.608 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39763.log 2024-09-16T23:43:15.608 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.48908.log: /var/log/ceph/cluster2-client.mirror.71462.log 2024-09-16T23:43:15.608 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48908.log.gz 2024-09-16T23:43:15.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38365.log 2024-09-16T23:43:15.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48605.log 2024-09-16T23:43:15.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38365.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71462.log.gz 2024-09-16T23:43:15.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43361.log 2024-09-16T23:43:15.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48605.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48605.log.gz 2024-09-16T23:43:15.626 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45069.log 2024-09-16T23:43:15.626 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60673.log/var/log/ceph/cluster2-client.mirror.43361.log: 2024-09-16T23:43:15.626 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43361.log.gz/var/log/ceph/cluster1-client.mirror.45069.log: 2024-09-16T23:43:15.626 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45069.log.gz 2024-09-16T23:43:15.626 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35684.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70270.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60673.log.gz 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35684.log.gz 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- 93.6% /var/log/ceph/cluster2-client.admin.37967.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46373.log.gz 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48421.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.37967.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70270.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68528.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48421.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69113.log 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68528.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68528.log.gz 2024-09-16T23:43:15.627 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73699.log 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% 70.8% -- replaced with /var/log/ceph/cluster2-client.admin.37967.log.gz 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39763.log.gz 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69113.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69113.log.gz 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60851.log 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73699.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73699.log.gz 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73596.log 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37020.log 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60851.log.gz 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63770.log 2024-09-16T23:43:15.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.37020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72141.log 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63770.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73596.log.gz 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48320.log 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.63770.log.gz 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43012.log 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72141.log: /var/log/ceph/cluster2-client.mirror.48320.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48320.log.gz 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72141.log.gz 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60535.log 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51695.log 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60535.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60535.log.gz 2024-09-16T23:43:15.629 INFO:teuthology.orchestra.run.smithi098.stderr: 73.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37020.log.gz 2024-09-16T23:43:15.634 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49885.log 2024-09-16T23:43:15.634 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51695.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51695.log.gz 2024-09-16T23:43:15.634 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43012.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48642.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49885.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49885.log.gz 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56449.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72432.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48642.log.gz 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48399.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72432.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56449.log.gz 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72432.log.gz 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64661.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68918.log 2024-09-16T23:43:15.635 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64661.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64661.log.gz 2024-09-16T23:43:15.636 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48399.log.gz 2024-09-16T23:43:15.640 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57160.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51164.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.68918.log.gz 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57160.log: gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57160.log.gz 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/cluster2-client.mirror.46474.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.71688.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46474.log: /var/log/ceph/cluster1-client.mirror.51164.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46474.log.gz 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51164.log.gz 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65882.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.0.71688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61511.log 2024-09-16T23:43:15.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65882.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65882.log.gz 2024-09-16T23:43:15.643 INFO:teuthology.orchestra.run.smithi098.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.71688.log.gz 2024-09-16T23:43:15.647 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61511.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61511.log.gz 2024-09-16T23:43:15.647 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49432.log 2024-09-16T23:43:15.647 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54560.log 2024-09-16T23:43:15.661 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49432.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46024.log.gz 2024-09-16T23:43:15.664 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45944.log 2024-09-16T23:43:15.665 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48622.log 2024-09-16T23:43:15.665 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45944.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54560.log.gz 2024-09-16T23:43:15.665 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63661.log 2024-09-16T23:43:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48622.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48622.log.gz 2024-09-16T23:43:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63265.log 2024-09-16T23:43:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63661.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63661.log.gz 2024-09-16T23:43:15.669 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47819.log 2024-09-16T23:43:15.669 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63265.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63265.log.gz 2024-09-16T23:43:15.669 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43872.log 2024-09-16T23:43:15.670 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47819.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47819.log.gz 2024-09-16T23:43:15.670 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70464.log 2024-09-16T23:43:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43872.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43872.log.gz 2024-09-16T23:43:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68343.log 2024-09-16T23:43:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70464.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70464.log.gz 2024-09-16T23:43:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40640.log 2024-09-16T23:43:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68343.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68343.log.gz 2024-09-16T23:43:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73995.log 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36572.log 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73995.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73995.log.gz 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72561.log 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.36572.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36572.log.gz 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58141.log 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72561.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.72561.log.gz 2024-09-16T23:43:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66662.log 2024-09-16T23:43:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58141.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58141.log.gz 2024-09-16T23:43:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63853.log 2024-09-16T23:43:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66662.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66662.log.gz 2024-09-16T23:43:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67270.log 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63853.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63853.log.gz 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50248.log 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67270.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67270.log.gz 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49868.log 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50248.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50248.log.gz 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41867.log 2024-09-16T23:43:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49868.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49868.log.gz 2024-09-16T23:43:15.688 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59820.log 2024-09-16T23:43:15.688 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41867.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41867.log.gz 2024-09-16T23:43:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64227.log 2024-09-16T23:43:15.690 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59820.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59820.log.gz 2024-09-16T23:43:15.690 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51779.log 2024-09-16T23:43:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64227.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64227.log.gz 2024-09-16T23:43:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70878.log 2024-09-16T23:43:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51779.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51779.log.gz 2024-09-16T23:43:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66190.log 2024-09-16T23:43:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70878.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70878.log.gz 2024-09-16T23:43:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63430.log 2024-09-16T23:43:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66190.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66190.log.gz 2024-09-16T23:43:15.696 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72510.log 2024-09-16T23:43:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63430.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63430.log.gz 2024-09-16T23:43:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39954.log 2024-09-16T23:43:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72510.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72510.log.gz 2024-09-16T23:43:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46331.log 2024-09-16T23:43:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65572.log 2024-09-16T23:43:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46331.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46331.log.gz 2024-09-16T23:43:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46351.log 2024-09-16T23:43:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65572.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65572.log.gz 2024-09-16T23:43:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46920.log 2024-09-16T23:43:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46351.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46351.log.gz 2024-09-16T23:43:15.704 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39954.log.gz 2024-09-16T23:43:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60743.log 2024-09-16T23:43:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46920.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46920.log.gz 2024-09-16T23:43:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46431.log 2024-09-16T23:43:15.709 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60743.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60743.log.gz 2024-09-16T23:43:15.709 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67315.log 2024-09-16T23:43:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46431.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46431.log.gz 2024-09-16T23:43:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44173.log 2024-09-16T23:43:15.712 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67315.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67315.log.gz 2024-09-16T23:43:15.712 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70984.log 2024-09-16T23:43:15.712 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40640.log.gz 2024-09-16T23:43:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44173.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44173.log.gz 2024-09-16T23:43:15.718 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67562.log 2024-09-16T23:43:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70984.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70984.log.gz 2024-09-16T23:43:15.719 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53696.log 2024-09-16T23:43:15.721 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67562.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67562.log.gz 2024-09-16T23:43:15.721 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69505.log 2024-09-16T23:43:15.722 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53696.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53696.log.gz 2024-09-16T23:43:15.722 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67382.log 2024-09-16T23:43:15.731 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66752.log 2024-09-16T23:43:15.733 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67382.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67382.log.gz 2024-09-16T23:43:15.746 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57319.log 2024-09-16T23:43:15.747 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66752.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66752.log.gz 2024-09-16T23:43:15.753 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37625.log 2024-09-16T23:43:15.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57319.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57319.log.gz 2024-09-16T23:43:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51227.log 2024-09-16T23:43:15.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.37625.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37625.log.gz 2024-09-16T23:43:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41543.log 2024-09-16T23:43:15.781 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51227.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51227.log.gz 2024-09-16T23:43:15.783 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48502.log.gz 2024-09-16T23:43:15.784 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57965.log 2024-09-16T23:43:15.785 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47550.log 2024-09-16T23:43:15.787 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57965.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57965.log.gz 2024-09-16T23:43:15.788 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65324.log 2024-09-16T23:43:15.789 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47550.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47550.log.gz 2024-09-16T23:43:15.789 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48871.log 2024-09-16T23:43:15.790 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65324.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65324.log.gz 2024-09-16T23:43:15.790 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55786.log 2024-09-16T23:43:15.791 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48871.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48871.log.gz 2024-09-16T23:43:15.791 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54518.log 2024-09-16T23:43:15.792 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55786.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55786.log.gz 2024-09-16T23:43:15.793 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58719.log 2024-09-16T23:43:15.794 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54518.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54518.log.gz 2024-09-16T23:43:15.794 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53206.log 2024-09-16T23:43:15.795 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58719.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58719.log.gz 2024-09-16T23:43:15.796 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46047.log 2024-09-16T23:43:15.797 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53206.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53206.log.gz 2024-09-16T23:43:15.797 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44365.log 2024-09-16T23:43:15.801 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46047.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46047.log.gz 2024-09-16T23:43:15.801 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42245.log 2024-09-16T23:43:15.804 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44365.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44365.log.gz 2024-09-16T23:43:15.806 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66347.log 2024-09-16T23:43:15.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42245.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42245.log.gz 2024-09-16T23:43:15.806 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64909.log 2024-09-16T23:43:15.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66347.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66347.log.gz 2024-09-16T23:43:15.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39718.log 2024-09-16T23:43:15.809 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64909.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64909.log.gz 2024-09-16T23:43:15.809 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49927.log 2024-09-16T23:43:15.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39718.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39602.log 2024-09-16T23:43:15.811 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49927.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39718.log.gz 2024-09-16T23:43:15.815 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41543.log.gz 2024-09-16T23:43:15.820 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48421.log.gz 2024-09-16T23:43:15.821 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71548.log 2024-09-16T23:43:15.822 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41681.log 2024-09-16T23:43:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71548.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71548.log.gz 2024-09-16T23:43:15.823 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60376.log 2024-09-16T23:43:15.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.41681.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41681.log.gz 2024-09-16T23:43:15.824 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61100.log 2024-09-16T23:43:15.825 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60376.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60376.log.gz 2024-09-16T23:43:15.826 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63554.log 2024-09-16T23:43:15.826 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61100.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61100.log.gz 2024-09-16T23:43:15.827 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68462.log 2024-09-16T23:43:15.828 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63554.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63554.log.gz 2024-09-16T23:43:15.829 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42581.log 2024-09-16T23:43:15.830 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39602.log.gz 2024-09-16T23:43:15.830 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68462.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68462.log.gz 2024-09-16T23:43:15.831 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.63931.log 2024-09-16T23:43:15.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42581.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42581.log.gz 2024-09-16T23:43:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68163.log 2024-09-16T23:43:15.844 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.2.63931.log: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.2.63931.log.gz 2024-09-16T23:43:15.845 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64496.log 2024-09-16T23:43:15.845 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68163.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68163.log.gz 2024-09-16T23:43:15.846 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60831.log 2024-09-16T23:43:15.846 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64496.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64496.log.gz 2024-09-16T23:43:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67337.log 2024-09-16T23:43:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60831.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60831.log.gz 2024-09-16T23:43:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49826.log 2024-09-16T23:43:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67337.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67337.log.gz 2024-09-16T23:43:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60151.log 2024-09-16T23:43:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49826.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49826.log.gz 2024-09-16T23:43:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57297.log 2024-09-16T23:43:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60151.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60151.log.gz 2024-09-16T23:43:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55363.log 2024-09-16T23:43:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57297.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57297.log.gz 2024-09-16T23:43:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40243.log 2024-09-16T23:43:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55363.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55363.log.gz 2024-09-16T23:43:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71247.log 2024-09-16T23:43:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40243.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45631.log 2024-09-16T23:43:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71247.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71247.log.gz 2024-09-16T23:43:15.858 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40243.log.gz 2024-09-16T23:43:15.858 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68940.log.gz 2024-09-16T23:43:15.859 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68635.log 2024-09-16T23:43:15.860 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45631.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45631.log.gz 2024-09-16T23:43:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72285.log 2024-09-16T23:43:15.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72917.log 2024-09-16T23:43:15.869 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68635.log.gz 2024-09-16T23:43:15.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55166.log 2024-09-16T23:43:15.870 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72285.log.gz 2024-09-16T23:43:15.870 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54231.log 2024-09-16T23:43:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55166.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55166.log.gz 2024-09-16T23:43:15.871 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72917.log.gz 2024-09-16T23:43:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59889.log 2024-09-16T23:43:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37557.log 2024-09-16T23:43:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59889.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54231.log.gz 2024-09-16T23:43:15.873 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59889.log.gz 2024-09-16T23:43:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67787.log 2024-09-16T23:43:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/cluster1-client.mirror.37557.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41771.log 2024-09-16T23:43:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47977.log 2024-09-16T23:43:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41771.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41771.log.gz 2024-09-16T23:43:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67787.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67787.log.gz 2024-09-16T23:43:15.876 INFO:teuthology.orchestra.run.smithi098.stderr: 63.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37557.log.gz 2024-09-16T23:43:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65406.log 2024-09-16T23:43:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47977.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47977.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.37136.log 2024-09-16T23:43:15.882 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73637.log 2024-09-16T23:43:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.3.37136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66415.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73637.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65406.log.gz 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73637.log.gz 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71046.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58228.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66415.log.gz 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71046.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64847.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58228.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71046.log.gz 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58228.log.gz 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37506.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65095.log 2024-09-16T23:43:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37506.log: /var/log/ceph/cluster2-client.mirror.64847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36362.log 2024-09-16T23:43:15.885 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65095.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64847.log.gz 2024-09-16T23:43:15.885 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/cluster2-client.admin.36362.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36362.log.gz 2024-09-16T23:43:15.885 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65095.log.gz 2024-09-16T23:43:15.885 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61197.log 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61197.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61197.log.gz 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59121.log 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68614.log 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71269.log.gz 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr: 62.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37506.log.gz 2024-09-16T23:43:15.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40503.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.68614.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59121.log.gz 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65799.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68614.log.gz 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40503.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46554.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56072.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63534.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46554.log.gz 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40503.log.gz 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56072.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65799.log.gz 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66595.log 2024-09-16T23:43:15.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63534.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63534.log.gz 2024-09-16T23:43:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64889.log 2024-09-16T23:43:15.894 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56072.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36656.log 2024-09-16T23:43:15.894 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.894 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66595.log: /var/log/ceph/cluster2-client.mirror.64889.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66595.log.gz 2024-09-16T23:43:15.894 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64889.log.gz 2024-09-16T23:43:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66775.log 2024-09-16T23:43:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39299.log 2024-09-16T23:43:15.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66775.log: /var/log/ceph/cluster1-client.admin.36656.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36656.log.gz 2024-09-16T23:43:15.895 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66775.log.gz 2024-09-16T23:43:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68570.log 2024-09-16T23:43:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68758.log 2024-09-16T23:43:15.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68570.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68570.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/cluster1-client.mirror.54138.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68758.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73617.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68758.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65489.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54138.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41454.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73617.log: /var/log/ceph/cluster2-client.mirror.65489.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73617.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65489.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45572.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57453.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61221.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40178.log 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57453.log.gz 2024-09-16T23:43:15.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72452.log 2024-09-16T23:43:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40178.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61221.log.gz 2024-09-16T23:43:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61012.log 2024-09-16T23:43:15.904 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45572.log.gz 2024-09-16T23:43:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55435.log 2024-09-16T23:43:15.904 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.61012.log: -- /var/log/ceph/cluster2-client.mirror.45552.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55435.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55435.log.gz 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61012.log.gz 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71227.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45552.log: /var/log/ceph/cluster2-client.mirror.72452.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45552.log.gz 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72452.log.gz 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47759.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53403.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71227.log.gz 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72223.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47759.log: /var/log/ceph/cluster1-client.mirror.53403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57564.log 2024-09-16T23:43:15.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72223.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53403.log.gz 2024-09-16T23:43:15.909 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72223.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64165.log 2024-09-16T23:43:15.909 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57564.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57564.log.gz 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66820.log 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65964.log 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66820.log: /var/log/ceph/cluster2-client.mirror.64165.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66820.log.gz 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40178.log.gz 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42677.log 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65964.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64165.log.gz 2024-09-16T23:43:15.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44723.log 2024-09-16T23:43:15.913 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.42677.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42677.log.gz 2024-09-16T23:43:15.913 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57011.log 2024-09-16T23:43:15.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44723.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44723.log.gz 2024-09-16T23:43:15.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.71306.log 2024-09-16T23:43:15.915 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65964.log.gz 2024-09-16T23:43:15.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57011.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57011.log.gz 2024-09-16T23:43:15.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62911.log 2024-09-16T23:43:15.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62911.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41454.log.gz 2024-09-16T23:43:15.917 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44322.log 2024-09-16T23:43:15.917 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.71306.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62911.log.gz 2024-09-16T23:43:15.920 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.71306.log.gz 2024-09-16T23:43:15.921 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50381.log 2024-09-16T23:43:15.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41566.log 2024-09-16T23:43:15.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50381.log: /var/log/ceph/cluster1-client.mirror.44322.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50381.log.gz 2024-09-16T23:43:15.926 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44322.log.gz 2024-09-16T23:43:15.927 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41608.log 2024-09-16T23:43:15.927 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41566.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42356.log 2024-09-16T23:43:15.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41608.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64310.log 2024-09-16T23:43:15.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42356.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42356.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66977.log 2024-09-16T23:43:15.930 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:15.931 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64310.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64310.log.gz 2024-09-16T23:43:15.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66977.log: 93.5%gzip -5 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66977.log.gz 2024-09-16T23:43:15.934 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.48743.log 2024-09-16T23:43:15.934 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39299.log.gz 2024-09-16T23:43:15.934 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62254.log 2024-09-16T23:43:15.934 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54897.log 2024-09-16T23:43:15.935 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62254.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62254.log.gz 2024-09-16T23:43:15.935 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73822.log 2024-09-16T23:43:15.936 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54897.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54897.log.gz 2024-09-16T23:43:15.936 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66392.log 2024-09-16T23:43:15.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73822.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73822.log.gz 2024-09-16T23:43:15.942 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57872.log 2024-09-16T23:43:15.946 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66392.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66392.log.gz 2024-09-16T23:43:15.946 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41112.log 2024-09-16T23:43:15.946 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57872.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57872.log.gz 2024-09-16T23:43:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61695.log 2024-09-16T23:43:15.947 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41112.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41608.log.gz 2024-09-16T23:43:15.948 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62994.log 2024-09-16T23:43:15.949 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61695.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61695.log.gz 2024-09-16T23:43:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68050.log 2024-09-16T23:43:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62994.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62994.log.gz 2024-09-16T23:43:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67517.log 2024-09-16T23:43:15.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68050.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68050.log.gz 2024-09-16T23:43:15.952 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58968.log 2024-09-16T23:43:15.952 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67517.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67517.log.gz 2024-09-16T23:43:15.953 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58968.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58968.log.gz 2024-09-16T23:43:15.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51899.log 2024-09-16T23:43:15.961 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.37136.log.gz 2024-09-16T23:43:15.965 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41112.log.gz 2024-09-16T23:43:15.969 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47799.log 2024-09-16T23:43:15.970 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51899.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51899.log.gz 2024-09-16T23:43:15.970 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49125.log 2024-09-16T23:43:15.971 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47799.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47799.log.gz 2024-09-16T23:43:15.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66167.log 2024-09-16T23:43:15.979 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.49125.log: 95.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49125.log.gz 2024-09-16T23:43:15.981 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42076.log 2024-09-16T23:43:15.981 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66167.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66167.log.gz 2024-09-16T23:43:15.982 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73760.log 2024-09-16T23:43:15.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.42076.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.42076.log.gz 2024-09-16T23:43:15.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55211.log 2024-09-16T23:43:15.985 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73760.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73760.log.gz 2024-09-16T23:43:15.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44560.log 2024-09-16T23:43:15.987 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55211.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55211.log.gz 2024-09-16T23:43:15.988 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39997.log 2024-09-16T23:43:15.994 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44560.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44560.log.gz 2024-09-16T23:43:15.994 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54210.log 2024-09-16T23:43:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57277.log 2024-09-16T23:43:15.995 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54210.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54210.log.gz 2024-09-16T23:43:15.997 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39997.log.gz 2024-09-16T23:43:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50313.log 2024-09-16T23:43:16.003 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57277.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57277.log.gz 2024-09-16T23:43:16.003 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57985.log 2024-09-16T23:43:16.007 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50313.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50313.log.gz 2024-09-16T23:43:16.007 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46084.log 2024-09-16T23:43:16.012 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57985.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49848.log.gz 2024-09-16T23:43:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43892.log 2024-09-16T23:43:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49088.log 2024-09-16T23:43:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40221.log 2024-09-16T23:43:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62703.log 2024-09-16T23:43:16.015 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46084.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46084.log.gz 2024-09-16T23:43:16.016 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.37135.log 2024-09-16T23:43:16.018 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62703.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62703.log.gz 2024-09-16T23:43:16.025 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4%/var/log/ceph/cluster2-client.mirror.49088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70964.log 2024-09-16T23:43:16.027 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57985.log.gz 2024-09-16T23:43:16.028 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43892.log.gz 2024-09-16T23:43:16.028 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71087.log 2024-09-16T23:43:16.031 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70964.log: 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70964.log.gz 2024-09-16T23:43:16.041 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40221.log: /var/log/ceph/cluster2-client.mirror.2.37135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40703.log 2024-09-16T23:43:16.042 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71087.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71087.log.gz 2024-09-16T23:43:16.048 INFO:teuthology.orchestra.run.smithi096.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-16T23:43:16.058 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62029.log 2024-09-16T23:43:16.058 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40221.log.gz 2024-09-16T23:43:16.058 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56339.log 2024-09-16T23:43:16.058 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62029.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62029.log.gz 2024-09-16T23:43:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53230.log 2024-09-16T23:43:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56339.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56339.log.gz 2024-09-16T23:43:16.060 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67900.log 2024-09-16T23:43:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53230.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53230.log.gz 2024-09-16T23:43:16.061 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65303.log 2024-09-16T23:43:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67900.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67900.log.gz 2024-09-16T23:43:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58290.log 2024-09-16T23:43:16.063 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65303.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65303.log.gz 2024-09-16T23:43:16.064 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73238.log 2024-09-16T23:43:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58290.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58290.log.gz 2024-09-16T23:43:16.065 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73781.log 2024-09-16T23:43:16.066 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73238.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73238.log.gz 2024-09-16T23:43:16.067 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39341.log 2024-09-16T23:43:16.067 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73781.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73781.log.gz 2024-09-16T23:43:16.068 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62121.log 2024-09-16T23:43:16.069 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32740.log 2024-09-16T23:43:16.070 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62121.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.39341.log.gz 2024-09-16T23:43:16.071 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62121.log.gz 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.37135.log.gz 2024-09-16T23:43:16.071 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.071 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39742.log 2024-09-16T23:43:16.071 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32740.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32740.log.gz 2024-09-16T23:43:16.073 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63703.log 2024-09-16T23:43:16.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44517.log 2024-09-16T23:43:16.075 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63703.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63703.log.gz 2024-09-16T23:43:16.076 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37933.log 2024-09-16T23:43:16.081 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44517.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44517.log.gz 2024-09-16T23:43:16.082 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49410.log 2024-09-16T23:43:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.37933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68186.log 2024-09-16T23:43:16.083 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49410.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49410.log.gz 2024-09-16T23:43:16.086 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39742.log.gz 2024-09-16T23:43:16.087 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40703.log.gz 2024-09-16T23:43:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56401.log 2024-09-16T23:43:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68186.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68186.log.gz 2024-09-16T23:43:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44474.log 2024-09-16T23:43:16.089 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56401.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56401.log.gz 2024-09-16T23:43:16.089 INFO:teuthology.orchestra.run.smithi098.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.37933.log.gz 2024-09-16T23:43:16.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62411.log 2024-09-16T23:43:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44474.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44474.log.gz 2024-09-16T23:43:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62479.log 2024-09-16T23:43:16.098 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62411.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41566.log.gz 2024-09-16T23:43:16.100 INFO:teuthology.orchestra.run.smithi098.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.62411.log.gz 2024-09-16T23:43:16.100 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62787.log 2024-09-16T23:43:16.101 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62479.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62479.log.gz 2024-09-16T23:43:16.102 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44107.log 2024-09-16T23:43:16.102 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62787.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62787.log.gz 2024-09-16T23:43:16.103 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59009.log 2024-09-16T23:43:16.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44107.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44107.log.gz 2024-09-16T23:43:16.109 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49029.log 2024-09-16T23:43:16.111 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59009.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59009.log.gz 2024-09-16T23:43:16.113 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48078.log 2024-09-16T23:43:16.113 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49029.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49029.log.gz 2024-09-16T23:43:16.113 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70567.log 2024-09-16T23:43:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57232.log 2024-09-16T23:43:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70567.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70567.log.gz 2024-09-16T23:43:16.138 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69790.log 2024-09-16T23:43:16.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57232.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57232.log.gz 2024-09-16T23:43:16.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70608.log 2024-09-16T23:43:16.147 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69790.log: 87.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57256.log 2024-09-16T23:43:16.147 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69790.log.gz 2024-09-16T23:43:16.147 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70608.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70608.log.gz 2024-09-16T23:43:16.161 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37431.log 2024-09-16T23:43:16.161 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57256.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57256.log.gz 2024-09-16T23:43:16.171 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45944.log.gz 2024-09-16T23:43:16.179 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50779.log 2024-09-16T23:43:16.179 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.37431.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37431.log.gz 2024-09-16T23:43:16.197 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46250.log 2024-09-16T23:43:16.198 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50779.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50779.log.gz 2024-09-16T23:43:16.211 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50558.log 2024-09-16T23:43:16.211 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46250.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46250.log.gz 2024-09-16T23:43:16.222 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63348.log 2024-09-16T23:43:16.223 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50558.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50558.log.gz 2024-09-16T23:43:16.230 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69505.log.gz 2024-09-16T23:43:16.231 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73895.log 2024-09-16T23:43:16.232 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63348.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63348.log.gz 2024-09-16T23:43:16.233 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48194.log 2024-09-16T23:43:16.233 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73895.log.gz 2024-09-16T23:43:16.233 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55253.log 2024-09-16T23:43:16.237 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.48194.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48194.log.gz 2024-09-16T23:43:16.237 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45131.log 2024-09-16T23:43:16.238 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55253.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55253.log.gz 2024-09-16T23:43:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44388.log 2024-09-16T23:43:16.239 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45131.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45131.log.gz 2024-09-16T23:43:16.240 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44044.log 2024-09-16T23:43:16.245 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44388.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44388.log.gz 2024-09-16T23:43:16.246 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63203.log 2024-09-16T23:43:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44044.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44044.log.gz 2024-09-16T23:43:16.252 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63286.log 2024-09-16T23:43:16.253 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63203.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63203.log.gz 2024-09-16T23:43:16.254 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57720.log 2024-09-16T23:43:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63286.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63286.log.gz 2024-09-16T23:43:16.255 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67855.log 2024-09-16T23:43:16.256 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57720.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47759.log.gz 2024-09-16T23:43:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67360.log 2024-09-16T23:43:16.258 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5%/var/log/ceph/cluster1-client.mirror.67855.log: -- replaced with /var/log/ceph/cluster2-client.mirror.57720.log.gz 2024-09-16T23:43:16.258 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67855.log.gz 2024-09-16T23:43:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53496.log 2024-09-16T23:43:16.258 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53965.log 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67360.log: 86.7%/var/log/ceph/cluster2-client.mirror.53496.log: -- replaced with /var/log/ceph/cluster1-client.mirror.67360.log.gz 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70203.log 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53965.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53965.log.gz 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54966.log 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70203.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53496.log.gz 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.70203.log.gz 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71148.log 2024-09-16T23:43:16.262 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64682.log 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71148.log: /var/log/ceph/cluster2-client.mirror.54966.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.71148.log.gz 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55476.log 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64682.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54966.log.gz 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64682.log.gz 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62616.log 2024-09-16T23:43:16.265 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63244.log 2024-09-16T23:43:16.266 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62616.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55476.log.gz 2024-09-16T23:43:16.268 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63244.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63244.log.gz 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62616.log.gz 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40041.log 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55540.log 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36024.log 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56007.log 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36024.log: 89.0% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36024.log.gz 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55540.log.gz 2024-09-16T23:43:16.271 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63014.log 2024-09-16T23:43:16.273 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52185.log 2024-09-16T23:43:16.273 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56007.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56007.log.gz/var/log/ceph/cluster2-client.mirror.63014.log: 2024-09-16T23:43:16.273 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39694.log 2024-09-16T23:43:16.274 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52185.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40041.log.gz 2024-09-16T23:43:16.275 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52185.log.gz 2024-09-16T23:43:16.275 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63014.log.gz 2024-09-16T23:43:16.277 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47146.log 2024-09-16T23:43:16.277 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39694.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54763.log 2024-09-16T23:43:16.277 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60332.log 2024-09-16T23:43:16.278 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54763.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54763.log.gz 2024-09-16T23:43:16.278 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37489.log 2024-09-16T23:43:16.279 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60332.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60332.log.gz 2024-09-16T23:43:16.281 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47920.log 2024-09-16T23:43:16.281 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.37489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47619.log 2024-09-16T23:43:16.286 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47920.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39694.log.gz 2024-09-16T23:43:16.287 INFO:teuthology.orchestra.run.smithi098.stderr: 70.4% -- replaced with /var/log/ceph/cluster1-client.admin.37489.log.gz 2024-09-16T23:43:16.292 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47619.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47619.log.gz 2024-09-16T23:43:16.294 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72035.log 2024-09-16T23:43:16.295 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49432.log.gz 2024-09-16T23:43:16.297 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67877.log 2024-09-16T23:43:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72035.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.72035.log.gz 2024-09-16T23:43:16.298 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62808.log 2024-09-16T23:43:16.299 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67877.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67877.log.gz 2024-09-16T23:43:16.299 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43629.log 2024-09-16T23:43:16.300 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62808.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62808.log.gz 2024-09-16T23:43:16.300 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65427.log 2024-09-16T23:43:16.301 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43629.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43629.log.gz 2024-09-16T23:43:16.301 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73258.log 2024-09-16T23:43:16.305 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65427.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65427.log.gz 2024-09-16T23:43:16.306 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69175.log 2024-09-16T23:43:16.306 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73258.log: 86.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73258.log.gz 2024-09-16T23:43:16.307 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56915.log 2024-09-16T23:43:16.307 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69175.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69175.log.gz 2024-09-16T23:43:16.308 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67112.log 2024-09-16T23:43:16.308 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56915.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56915.log.gz 2024-09-16T23:43:16.309 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49756.log 2024-09-16T23:43:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67112.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67112.log.gz 2024-09-16T23:43:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41153.log 2024-09-16T23:43:16.332 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59234.log 2024-09-16T23:43:16.342 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41153.log: /var/log/ceph/cluster1-client.mirror.59234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57405.log 2024-09-16T23:43:16.351 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50602.log 2024-09-16T23:43:16.355 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57405.log: -- replaced with /var/log/ceph/cluster1-client.mirror.59234.log.gz 2024-09-16T23:43:16.364 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57405.log.gz 2024-09-16T23:43:16.367 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50602.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50602.log.gz 2024-09-16T23:43:16.368 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36236.log 2024-09-16T23:43:16.377 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58049.log 2024-09-16T23:43:16.378 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36236.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41153.log.gz 2024-09-16T23:43:16.378 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36236.log.gz 2024-09-16T23:43:16.386 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66460.log 2024-09-16T23:43:16.387 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58049.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58049.log.gz 2024-09-16T23:43:16.388 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64103.log 2024-09-16T23:43:16.388 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66460.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66460.log.gz 2024-09-16T23:43:16.397 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42943.log 2024-09-16T23:43:16.406 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64103.log: 90.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72605.log 2024-09-16T23:43:16.412 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42943.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64103.log.gz 2024-09-16T23:43:16.415 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66527.log 2024-09-16T23:43:16.417 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72605.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72605.log.gz 2024-09-16T23:43:16.419 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42943.log.gz 2024-09-16T23:43:16.421 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48078.log.gz 2024-09-16T23:43:16.422 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51024.log 2024-09-16T23:43:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66527.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66527.log.gz 2024-09-16T23:43:16.423 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49927.log.gz 2024-09-16T23:43:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68298.log 2024-09-16T23:43:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52452.log 2024-09-16T23:43:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68298.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.51024.log.gz 2024-09-16T23:43:16.426 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68298.log.gz 2024-09-16T23:43:16.426 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.50073.log 2024-09-16T23:43:16.426 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46533.log 2024-09-16T23:43:16.426 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52452.log.gz 2024-09-16T23:43:16.426 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.3.50073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73945.log 2024-09-16T23:43:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46533.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.50073.log.gz 2024-09-16T23:43:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52759.log 2024-09-16T23:43:16.429 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73945.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73945.log.gz 2024-09-16T23:43:16.430 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59322.log 2024-09-16T23:43:16.431 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52759.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52759.log.gz 2024-09-16T23:43:16.431 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49471.log 2024-09-16T23:43:16.431 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49088.log.gz 2024-09-16T23:43:16.433 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59322.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59322.log.gz 2024-09-16T23:43:16.433 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44580.log 2024-09-16T23:43:16.433 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53624.log 2024-09-16T23:43:16.433 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44580.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49471.log.gz 2024-09-16T23:43:16.434 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54496.log 2024-09-16T23:43:16.435 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53624.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53624.log.gz 2024-09-16T23:43:16.435 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60352.log 2024-09-16T23:43:16.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54496.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54496.log.gz 2024-09-16T23:43:16.437 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55315.log 2024-09-16T23:43:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60352.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60352.log.gz 2024-09-16T23:43:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73487.log 2024-09-16T23:43:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55315.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44580.log.gz 2024-09-16T23:43:16.439 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55315.log.gz 2024-09-16T23:43:16.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54700.log 2024-09-16T23:43:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73487.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73487.log.gz 2024-09-16T23:43:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52165.log 2024-09-16T23:43:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54700.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56891.log 2024-09-16T23:43:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52165.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54700.log.gz 2024-09-16T23:43:16.442 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70441.log 2024-09-16T23:43:16.443 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56891.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56891.log.gz 2024-09-16T23:43:16.443 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52165.log.gz 2024-09-16T23:43:16.444 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51962.log 2024-09-16T23:43:16.444 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62952.log 2024-09-16T23:43:16.444 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70441.log: /var/log/ceph/cluster2-client.mirror.51962.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70441.log.gz 2024-09-16T23:43:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51962.log.gz 2024-09-16T23:43:16.445 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61961.log 2024-09-16T23:43:16.445 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47370.log 2024-09-16T23:43:16.446 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62952.log: /var/log/ceph/cluster1-client.mirror.61961.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62952.log.gz 2024-09-16T23:43:16.447 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47370.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47370.log.gz 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61961.log.gz 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74290.log 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44216.log 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62932.log 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74290.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74290.log.gz 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52831.log 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62932.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62932.log.gz 2024-09-16T23:43:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64599.log 2024-09-16T23:43:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52831.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52831.log.gz 2024-09-16T23:43:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72752.log 2024-09-16T23:43:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64599.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64599.log.gz 2024-09-16T23:43:16.454 INFO:teuthology.orchestra.run.smithi098.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38365.log.gz 2024-09-16T23:43:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51289.log 2024-09-16T23:43:16.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72752.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70183.log 2024-09-16T23:43:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51289.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72752.log.gz 2024-09-16T23:43:16.455 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51289.log.gz 2024-09-16T23:43:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59077.log 2024-09-16T23:43:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72793.log 2024-09-16T23:43:16.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70183.log: /var/log/ceph/cluster2-client.mirror.59077.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70183.log.gz 2024-09-16T23:43:16.457 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59077.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59481.log 2024-09-16T23:43:16.457 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.458 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44216.log.gz 2024-09-16T23:43:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72793.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72793.log.gz 2024-09-16T23:43:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44087.log 2024-09-16T23:43:16.458 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72182.log 2024-09-16T23:43:16.459 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59481.log: /var/log/ceph/cluster2-client.mirror.44087.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59481.log.gz 2024-09-16T23:43:16.459 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61941.log 2024-09-16T23:43:16.460 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49330.log 2024-09-16T23:43:16.460 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72182.log: /var/log/ceph/cluster2-client.mirror.61941.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72182.log.gz 2024-09-16T23:43:16.461 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71483.log 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49330.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49330.log.gz 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61941.log.gz 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50073.log 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43721.log 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50073.log: /var/log/ceph/cluster1-client.mirror.71483.log: 81.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50073.log.gz 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71483.log.gz 2024-09-16T23:43:16.463 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73070.log 2024-09-16T23:43:16.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53600.log 2024-09-16T23:43:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43721.log: /var/log/ceph/cluster2-client.mirror.73070.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44087.log.gz 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43721.log.gz 2024-09-16T23:43:16.465 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71025.log 2024-09-16T23:43:16.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.53600.log: -- /var/log/ceph/cluster2-osd.2.log 2024-09-16T23:43:16.465 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.71025.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53600.log.gz 2024-09-16T23:43:16.468 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71025.log.gz 2024-09-16T23:43:16.468 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73070.log.gz 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50474.log 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65990.log 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59097.log 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52120.log 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.65990.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.65990.log.gz 2024-09-16T23:43:16.469 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50474.log.gz 2024-09-16T23:43:16.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59097.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59097.log.gz 2024-09-16T23:43:16.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.52120.log: --verbose -- /var/log/ceph/cluster1-client.mirror.59571.log 2024-09-16T23:43:16.476 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71441.log 2024-09-16T23:43:16.476 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52120.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66011.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59571.log: /var/log/ceph/cluster1-client.mirror.71441.log: 87.3% 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.71441.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.59571.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45614.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72100.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66011.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67652.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72100.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45614.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55050.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72100.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67652.log: gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67652.log.gz 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/cluster1-client.mirror.62595.log 2024-09-16T23:43:16.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72305.log 2024-09-16T23:43:16.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62595.log: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46862.log 2024-09-16T23:43:16.478 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55050.log.gz 2024-09-16T23:43:16.478 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57848.log 2024-09-16T23:43:16.478 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72305.log: /var/log/ceph/cluster2-client.mirror.46862.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72305.log.gz 2024-09-16T23:43:16.478 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62595.log.gz 2024-09-16T23:43:16.483 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39094.log 2024-09-16T23:43:16.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53672.log 2024-09-16T23:43:16.483 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57848.log.gz 2024-09-16T23:43:16.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39094.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35942.log 2024-09-16T23:43:16.483 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53672.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39094.log.gz 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53672.log.gz 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51759.log 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35942.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32527.log 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.51759.log: -- /var/log/ceph/cluster2-client.mirror.65840.log 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32527.log: 88.9% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32527.log.gz 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.51759.log.gz 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56521.log 2024-09-16T23:43:16.484 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59459.log 2024-09-16T23:43:16.486 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65840.log.gz 2024-09-16T23:43:16.486 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56521.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56521.log.gz 2024-09-16T23:43:16.486 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60493.log 2024-09-16T23:43:16.486 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37608.log 2024-09-16T23:43:16.487 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60493.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59459.log.gz 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64620.log 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37608.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60493.log.gz 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56670.log 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54921.log 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64620.log.gz 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42740.log 2024-09-16T23:43:16.492 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6%/var/log/ceph/cluster1-client.mirror.54921.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56670.log.gz 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54921.log.gz 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55387.log 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60448.log 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42740.log.gz 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49714.log 2024-09-16T23:43:16.493 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55387.log.gz 2024-09-16T23:43:16.494 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60448.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60448.log.gz 2024-09-16T23:43:16.494 INFO:teuthology.orchestra.run.smithi098.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37608.log.gz 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69920.log 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52024.log 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69920.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49714.log.gz 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69920.log.gz 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62724.log 2024-09-16T23:43:16.497 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52024.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52024.log.gz 2024-09-16T23:43:16.501 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74064.log 2024-09-16T23:43:16.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37136.log 2024-09-16T23:43:16.501 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62724.log.gz 2024-09-16T23:43:16.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.74064.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74064.log.gz 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41502.log 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42139.log 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70506.log 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41502.log: /var/log/ceph/cluster1-client.mirror.42139.log: 74.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74040.log 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42139.log.gz 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27828.log 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70506.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70506.log.gz 2024-09-16T23:43:16.502 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74040.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74040.log.gz 2024-09-16T23:43:16.503 INFO:teuthology.orchestra.run.smithi098.stderr: 82.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37136.log.gz 2024-09-16T23:43:16.505 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49009.log 2024-09-16T23:43:16.505 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.tmp-client.admin.27828.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51827.log 2024-09-16T23:43:16.505 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.27828.log.gz 2024-09-16T23:43:16.506 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/cluster2-client.mirror.49009.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49635.log 2024-09-16T23:43:16.506 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51827.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51827.log.gz 2024-09-16T23:43:16.506 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.41502.log.gz 2024-09-16T23:43:16.508 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74374.log 2024-09-16T23:43:16.509 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49635.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49635.log.gz 2024-09-16T23:43:16.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64930.log 2024-09-16T23:43:16.514 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74374.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74374.log.gz 2024-09-16T23:43:16.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.58606.log 2024-09-16T23:43:16.515 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64930.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64930.log.gz 2024-09-16T23:43:16.515 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52655.log 2024-09-16T23:43:16.522 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.1.58606.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48743.log.gz 2024-09-16T23:43:16.523 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52879.log 2024-09-16T23:43:16.525 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52655.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52655.log.gz 2024-09-16T23:43:16.526 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45762.log 2024-09-16T23:43:16.527 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52879.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52879.log.gz 2024-09-16T23:43:16.527 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67697.log 2024-09-16T23:43:16.528 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45762.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45762.log.gz 2024-09-16T23:43:16.528 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66370.log 2024-09-16T23:43:16.529 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67697.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67697.log.gz 2024-09-16T23:43:16.529 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65033.log 2024-09-16T23:43:16.531 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66370.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66370.log.gz 2024-09-16T23:43:16.531 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68963.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65033.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.58606.log.gz 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65033.log.gz 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69878.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68963.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47274.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68963.log.gz 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69878.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69878.log.gz 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38052.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73049.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38052.log: /var/log/ceph/cluster2-client.mirror.47274.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47274.log.gz 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53893.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67810.log 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73049.log.gz 2024-09-16T23:43:16.538 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53893.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53893.log.gz 2024-09-16T23:43:16.539 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73419.log 2024-09-16T23:43:16.539 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67810.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67810.log.gz 2024-09-16T23:43:16.539 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47353.log 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73419.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73419.log.gz 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.38365.log 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47353.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47353.log.gz 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73461.log 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.1.38365.log: 63.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38052.log.gz 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46106.log 2024-09-16T23:43:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66910.log 2024-09-16T23:43:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66280.log 2024-09-16T23:43:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66910.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66910.log.gz 2024-09-16T23:43:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56276.log 2024-09-16T23:43:16.574 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66280.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66280.log.gz 2024-09-16T23:43:16.590 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67045.log 2024-09-16T23:43:16.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56276.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56276.log.gz 2024-09-16T23:43:16.598 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39806.log 2024-09-16T23:43:16.599 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67045.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67045.log.gz 2024-09-16T23:43:16.604 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47920.log.gz 2024-09-16T23:43:16.605 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54945.log 2024-09-16T23:43:16.606 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46997.log 2024-09-16T23:43:16.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54945.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54945.log.gz 2024-09-16T23:43:16.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53648.log 2024-09-16T23:43:16.611 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.46997.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46997.log.gz 2024-09-16T23:43:16.611 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67945.log 2024-09-16T23:43:16.612 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53648.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53648.log.gz 2024-09-16T23:43:16.612 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52269.log 2024-09-16T23:43:16.614 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67945.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67945.log.gz 2024-09-16T23:43:16.614 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63744.log 2024-09-16T23:43:16.619 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52269.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52269.log.gz 2024-09-16T23:43:16.619 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72732.log 2024-09-16T23:43:16.619 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63744.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63744.log.gz 2024-09-16T23:43:16.619 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62766.log 2024-09-16T23:43:16.620 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72732.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72732.log.gz 2024-09-16T23:43:16.620 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74088.log 2024-09-16T23:43:16.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62766.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62766.log.gz 2024-09-16T23:43:16.621 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68799.log 2024-09-16T23:43:16.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74088.log: 28.9% -- replaced with /var/log/ceph/cluster2-client.admin.74088.log.gz 2024-09-16T23:43:16.624 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37889.log 2024-09-16T23:43:16.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68799.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68799.log.gz 2024-09-16T23:43:16.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64475.log 2024-09-16T23:43:16.625 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.37889.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37889.log.gz 2024-09-16T23:43:16.625 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53110.log 2024-09-16T23:43:16.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64475.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64475.log.gz 2024-09-16T23:43:16.627 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.37020.log 2024-09-16T23:43:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53110.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53110.log.gz 2024-09-16T23:43:16.628 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56963.log 2024-09-16T23:43:16.629 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.2.37020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52365.log 2024-09-16T23:43:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56963.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56963.log.gz 2024-09-16T23:43:16.630 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39826.log 2024-09-16T23:43:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52365.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52365.log.gz 2024-09-16T23:43:16.632 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42703.log 2024-09-16T23:43:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58338.log 2024-09-16T23:43:16.633 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42703.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42703.log.gz 2024-09-16T23:43:16.639 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44064.log 2024-09-16T23:43:16.640 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58338.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58338.log.gz 2024-09-16T23:43:16.641 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55118.log 2024-09-16T23:43:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44064.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44064.log.gz 2024-09-16T23:43:16.647 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65820.log 2024-09-16T23:43:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55118.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55118.log.gz 2024-09-16T23:43:16.648 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58764.log 2024-09-16T23:43:16.650 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65820.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65820.log.gz 2024-09-16T23:43:16.651 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53869.log 2024-09-16T23:43:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58764.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58764.log.gz 2024-09-16T23:43:16.653 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36404.log 2024-09-16T23:43:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53869.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53869.log.gz 2024-09-16T23:43:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56473.log 2024-09-16T23:43:16.656 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.37020.log.gz 2024-09-16T23:43:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36404.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36404.log.gz 2024-09-16T23:43:16.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52920.log 2024-09-16T23:43:16.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56473.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56473.log.gz 2024-09-16T23:43:16.658 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67000.log 2024-09-16T23:43:16.660 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52920.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52920.log.gz 2024-09-16T23:43:16.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50695.log 2024-09-16T23:43:16.661 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67000.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67000.log.gz 2024-09-16T23:43:16.662 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63162.log 2024-09-16T23:43:16.664 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50695.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50695.log.gz 2024-09-16T23:43:16.664 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73340.log 2024-09-16T23:43:16.667 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63162.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63162.log.gz 2024-09-16T23:43:16.667 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56718.log 2024-09-16T23:43:16.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73340.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73340.log.gz 2024-09-16T23:43:16.671 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64806.log 2024-09-16T23:43:16.671 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56718.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56718.log.gz 2024-09-16T23:43:16.671 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70043.log 2024-09-16T23:43:16.672 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64806.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64806.log.gz 2024-09-16T23:43:16.672 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52957.log 2024-09-16T23:43:16.673 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70043.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70043.log.gz 2024-09-16T23:43:16.673 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71005.log 2024-09-16T23:43:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52957.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52957.log.gz 2024-09-16T23:43:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59772.log 2024-09-16T23:43:16.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71005.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71005.log.gz 2024-09-16T23:43:16.680 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65013.log 2024-09-16T23:43:16.680 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59772.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59772.log.gz 2024-09-16T23:43:16.681 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56497.log 2024-09-16T23:43:16.683 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65013.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65013.log.gz 2024-09-16T23:43:16.683 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58117.log 2024-09-16T23:43:16.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56497.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56497.log.gz 2024-09-16T23:43:16.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44743.log 2024-09-16T23:43:16.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58117.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58117.log.gz 2024-09-16T23:43:16.688 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40302.log 2024-09-16T23:43:16.689 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44743.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44743.log.gz 2024-09-16T23:43:16.690 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67675.log 2024-09-16T23:43:16.691 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46862.log.gz 2024-09-16T23:43:16.691 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.40302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57032.log 2024-09-16T23:43:16.691 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67675.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67675.log.gz 2024-09-16T23:43:16.692 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.57032.log: /var/log/ceph/cluster1-client.mirror.56317.log 2024-09-16T23:43:16.694 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57032.log.gz 2024-09-16T23:43:16.695 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39806.log.gz 2024-09-16T23:43:16.698 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42245.log 2024-09-16T23:43:16.699 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56317.log.gz 2024-09-16T23:43:16.699 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73658.log 2024-09-16T23:43:16.699 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55498.log 2024-09-16T23:43:16.699 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73658.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73658.log.gz 2024-09-16T23:43:16.700 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.0.42245.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40302.log.gz 2024-09-16T23:43:16.700 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66235.log 2024-09-16T23:43:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55498.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55498.log.gz 2024-09-16T23:43:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44683.log 2024-09-16T23:43:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66235.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66235.log.gz 2024-09-16T23:43:16.702 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43167.log 2024-09-16T23:43:16.704 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44683.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44683.log.gz 2024-09-16T23:43:16.704 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.43230.log 2024-09-16T23:43:16.704 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.43167.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.43167.log.gz 2024-09-16T23:43:16.705 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39826.log.gz 2024-09-16T23:43:16.707 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42245.log.gz 2024-09-16T23:43:16.707 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43697.log 2024-09-16T23:43:16.710 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.2.43230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62547.log 2024-09-16T23:43:16.720 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43697.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43697.log.gz 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45046.log 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62547.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49756.log.gz 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62547.log.gz 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65861.log 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36256.log 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65861.log: /var/log/ceph/cluster1-client.mirror.45046.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45046.log.gz 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65861.log.gz 2024-09-16T23:43:16.725 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44923.log 2024-09-16T23:43:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36256.log.gz 2024-09-16T23:43:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46511.log 2024-09-16T23:43:16.726 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44923.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44923.log.gz 2024-09-16T23:43:16.727 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58206.log 2024-09-16T23:43:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55231.log 2024-09-16T23:43:16.728 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.58206.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46511.log.gz 2024-09-16T23:43:16.728 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44763.log 2024-09-16T23:43:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55231.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58206.log.gz 2024-09-16T23:43:16.730 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55231.log.gz 2024-09-16T23:43:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74626.log 2024-09-16T23:43:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37105.log 2024-09-16T23:43:16.730 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74626.log: /var/log/ceph/cluster2-client.mirror.44763.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74626.log.gz 2024-09-16T23:43:16.731 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.1.37105.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44763.log.gz 2024-09-16T23:43:16.731 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39045.log 2024-09-16T23:43:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71210.log 2024-09-16T23:43:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60722.log 2024-09-16T23:43:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73870.log 2024-09-16T23:43:16.737 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60722.log: /var/log/ceph/cluster2-client.mirror.71210.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71210.log.gz 2024-09-16T23:43:16.737 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60722.log.gz 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62433.log 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61242.log 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62433.log: /var/log/ceph/cluster1-client.mirror.73870.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73870.log.gz 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39928.log 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.62433.log.gz 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55627.log 2024-09-16T23:43:16.738 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61242.log: /var/log/ceph/cluster1-client.admin.39928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73439.log 2024-09-16T23:43:16.739 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55627.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61242.log.gz 2024-09-16T23:43:16.739 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41064.log 2024-09-16T23:43:16.739 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73439.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73439.log.gz 2024-09-16T23:43:16.740 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46533.log.gz 2024-09-16T23:43:16.741 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40595.log 2024-09-16T23:43:16.742 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38018.log 2024-09-16T23:43:16.742 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41064.log.gz 2024-09-16T23:43:16.743 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52144.log 2024-09-16T23:43:16.744 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38018.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39928.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48684.log 2024-09-16T23:43:16.744 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52144.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52144.log.gz 2024-09-16T23:43:16.753 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40595.log.gz 2024-09-16T23:43:16.753 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38476.log 2024-09-16T23:43:16.753 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49598.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38476.log: /var/log/ceph/cluster2-client.mirror.48684.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48684.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42895.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49598.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61896.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42895.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42895.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45804.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61896.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38018.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61896.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62746.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45804.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45804.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35858.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62746.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62746.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53427.log 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35858.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35858.log.gz 2024-09-16T23:43:16.754 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32660.log 2024-09-16T23:43:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53427.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53427.log.gz 2024-09-16T23:43:16.756 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54986.log 2024-09-16T23:43:16.757 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32660.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32660.log.gz 2024-09-16T23:43:16.757 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48300.log 2024-09-16T23:43:16.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54986.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54986.log.gz 2024-09-16T23:43:16.758 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64289.log 2024-09-16T23:43:16.759 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48300.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48300.log.gz 2024-09-16T23:43:16.760 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69940.log 2024-09-16T23:43:16.762 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64289.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64289.log.gz 2024-09-16T23:43:16.763 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73091.log 2024-09-16T23:43:16.763 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69940.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69940.log.gz 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46662.log 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73091.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73091.log.gz 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70022.log 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46662.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46662.log.gz 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41751.log 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70022.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70022.log.gz 2024-09-16T23:43:16.770 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-16T23:43:16.771 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41751.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41751.log.gz 2024-09-16T23:43:16.771 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58872.log 2024-09-16T23:43:16.771 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38108.log 2024-09-16T23:43:16.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58872.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58872.log.gz 2024-09-16T23:43:16.773 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69961.log 2024-09-16T23:43:16.774 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63224.log 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38108.log.gz 2024-09-16T23:43:16.774 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.774 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69961.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69961.log.gz 2024-09-16T23:43:16.774 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45922.log 2024-09-16T23:43:16.776 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63224.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63224.log.gz 2024-09-16T23:43:16.776 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.49125.log 2024-09-16T23:43:16.776 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45922.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45922.log.gz 2024-09-16T23:43:16.776 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47530.log 2024-09-16T23:43:16.785 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.49125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44001.log 2024-09-16T23:43:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47530.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47530.log.gz 2024-09-16T23:43:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74206.log 2024-09-16T23:43:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44001.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62145.log 2024-09-16T23:43:16.810 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44001.log.gz 2024-09-16T23:43:16.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74206.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74206.log.gz 2024-09-16T23:43:16.815 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39045.log.gz 2024-09-16T23:43:16.820 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-16T23:43:16.822 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47146.log.gz 2024-09-16T23:43:16.823 INFO:teuthology.orchestra.run.smithi098.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55627.log.gz 2024-09-16T23:43:16.824 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61148.log 2024-09-16T23:43:16.826 INFO:teuthology.orchestra.run.smithi098.stderr:gzip/var/log/ceph/cluster1-client.mirror.62145.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56228.log 2024-09-16T23:43:16.827 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62145.log.gz 2024-09-16T23:43:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44903.log 2024-09-16T23:43:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56228.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56228.log.gz 2024-09-16T23:43:16.827 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42424.log 2024-09-16T23:43:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44903.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44903.log.gz 2024-09-16T23:43:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48950.log 2024-09-16T23:43:16.835 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61148.log: /var/log/ceph/cluster1-client.mirror.42424.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61148.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42424.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44823.log 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63575.log 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48950.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50982.log 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44823.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63575.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63575.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/cluster2-client.mirror.45111.log 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50982.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50982.log.gz 2024-09-16T23:43:16.836 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54186.log 2024-09-16T23:43:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45111.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45111.log.gz 2024-09-16T23:43:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62166.log 2024-09-16T23:43:16.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42376.log 2024-09-16T23:43:16.837 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54186.log.gz/var/log/ceph/cluster2-client.mirror.62166.log: 2024-09-16T23:43:16.838 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62166.log.gz 2024-09-16T23:43:16.838 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71189.log 2024-09-16T23:43:16.839 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54006.log 2024-09-16T23:43:16.839 INFO:teuthology.orchestra.run.smithi098.stderr: 89.1%/var/log/ceph/cluster2-client.mirror.71189.log: -- replaced with /var/log/ceph/cluster2-client.mirror.42376.log.gz 2024-09-16T23:43:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39206.log 2024-09-16T23:43:16.840 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54006.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71189.log.gz 2024-09-16T23:43:16.841 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54628.log 2024-09-16T23:43:16.841 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46394.log 2024-09-16T23:43:16.842 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54628.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54628.log.gz 2024-09-16T23:43:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58165.log 2024-09-16T23:43:16.843 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46394.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46394.log.gz 2024-09-16T23:43:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39206.log.gz 2024-09-16T23:43:16.845 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/cluster2-client.mirror.71527.log 2024-09-16T23:43:16.845 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54006.log.gz 2024-09-16T23:43:16.845 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58165.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58165.log.gz 2024-09-16T23:43:16.846 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64537.log 2024-09-16T23:43:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71527.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71527.log.gz 2024-09-16T23:43:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47999.log 2024-09-16T23:43:16.847 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40547.log 2024-09-16T23:43:16.848 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47999.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64537.log.gz 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.2.43230.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69855.log 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43097.log 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48987.log 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69855.log.gz 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46411.log 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48987.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48987.log.gz 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40547.log.gz 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43097.log.gz 2024-09-16T23:43:16.854 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49491.log 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46411.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65282.log 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49491.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46411.log.gz 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49491.log.gz 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57804.log 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65158.log 2024-09-16T23:43:16.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57804.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57804.log.gz 2024-09-16T23:43:16.867 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65282.log.gz 2024-09-16T23:43:16.867 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48056.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60992.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48056.log: /var/log/ceph/cluster2-client.mirror.65158.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48056.log.gz 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70811.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60992.log: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.49125.log.gz 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65158.log.gz 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52783.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36022.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60992.log.gz 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66100.log 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52783.log.gz/var/log/ceph/cluster2-client.admin.36022.log: 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36022.log.gz 2024-09-16T23:43:16.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67630.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66100.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66100.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61398.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67630.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67630.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41955.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61398.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61398.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55339.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45965.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55339.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55339.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43457.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45965.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45965.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.63051.log 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43457.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43457.log.gz 2024-09-16T23:43:16.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47513.log 2024-09-16T23:43:16.870 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41955.log.gz 2024-09-16T23:43:16.874 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.1.63051.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.63051.log.gz 2024-09-16T23:43:16.874 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44843.log 2024-09-16T23:43:16.875 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47513.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47513.log.gz 2024-09-16T23:43:16.876 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69425.log 2024-09-16T23:43:16.876 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44843.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72691.log 2024-09-16T23:43:16.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69425.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69425.log.gz 2024-09-16T23:43:16.879 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44843.log.gz 2024-09-16T23:43:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54090.log 2024-09-16T23:43:16.880 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72691.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74416.log 2024-09-16T23:43:16.880 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72691.log.gz 2024-09-16T23:43:16.881 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54090.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54090.log.gz 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69028.log 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68253.log 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74416.log.gz 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54825.log 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68253.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69028.log.gz 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68253.log.gz 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64434.log 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54825.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54825.log.gz 2024-09-16T23:43:16.887 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42497.log 2024-09-16T23:43:16.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64434.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64434.log.gz 2024-09-16T23:43:16.890 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64785.log 2024-09-16T23:43:16.890 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42497.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42497.log.gz 2024-09-16T23:43:16.890 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58806.log 2024-09-16T23:43:16.890 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65075.log 2024-09-16T23:43:16.891 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64785.log: /var/log/ceph/cluster2-client.mirror.58806.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64785.log.gz 2024-09-16T23:43:16.893 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58806.log.gz 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74332.log 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60785.log 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65075.log.gz 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74332.log.gz 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.32700.log 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46900.log 2024-09-16T23:43:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.32700.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.32700.log.gz 2024-09-16T23:43:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49513.log 2024-09-16T23:43:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46900.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46900.log.gz 2024-09-16T23:43:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57740.log 2024-09-16T23:43:16.904 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60785.log: /var/log/ceph/cluster2-client.mirror.49513.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60785.log.gz 2024-09-16T23:43:16.904 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63451.log 2024-09-16T23:43:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57740.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57740.log.gz 2024-09-16T23:43:16.909 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62346.log 2024-09-16T23:43:16.914 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63451.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63451.log.gz 2024-09-16T23:43:16.914 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64641.log 2024-09-16T23:43:16.915 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62346.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62346.log.gz 2024-09-16T23:43:16.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54431.log 2024-09-16T23:43:16.923 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64641.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64641.log.gz 2024-09-16T23:43:16.923 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48379.log 2024-09-16T23:43:16.924 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54431.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54431.log.gz 2024-09-16T23:43:16.925 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56136.log 2024-09-16T23:43:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48379.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48379.log.gz 2024-09-16T23:43:16.928 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57896.log 2024-09-16T23:43:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56136.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56136.log.gz 2024-09-16T23:43:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50185.log 2024-09-16T23:43:16.930 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57896.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57896.log.gz 2024-09-16T23:43:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48664.log 2024-09-16T23:43:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50185.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50185.log.gz 2024-09-16T23:43:16.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49293.log 2024-09-16T23:43:16.940 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53293.log 2024-09-16T23:43:16.951 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49293.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49009.log.gz 2024-09-16T23:43:16.951 INFO:teuthology.orchestra.run.smithi098.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49293.log.gz 2024-09-16T23:43:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71838.log 2024-09-16T23:43:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53293.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53293.log.gz 2024-09-16T23:43:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37950.log 2024-09-16T23:43:16.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71838.log: 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71838.log.gz 2024-09-16T23:43:16.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49677.log 2024-09-16T23:43:16.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59868.log 2024-09-16T23:43:16.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49677.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.37950.log.gz 2024-09-16T23:43:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69343.log 2024-09-16T23:43:16.973 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59868.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59868.log.gz 2024-09-16T23:43:16.980 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37105.log.gz 2024-09-16T23:43:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51092.log 2024-09-16T23:43:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69343.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.69343.log.gz 2024-09-16T23:43:16.994 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36740.log 2024-09-16T23:43:17.008 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66955.log 2024-09-16T23:43:17.009 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36740.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36740.log.gz 2024-09-16T23:43:17.009 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51092.log.gz 2024-09-16T23:43:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51361.log 2024-09-16T23:43:17.026 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66955.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66955.log.gz 2024-09-16T23:43:17.070 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49598.log.gz 2024-09-16T23:43:17.070 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72876.log 2024-09-16T23:43:17.071 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51361.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51361.log.gz 2024-09-16T23:43:17.071 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72343.log 2024-09-16T23:43:17.071 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72876.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72876.log.gz 2024-09-16T23:43:17.071 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59169.log 2024-09-16T23:43:17.071 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72343.log: 84.1% -- replaced with /var/log/ceph/cluster1-client.mirror.72343.log.gz 2024-09-16T23:43:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41294.log 2024-09-16T23:43:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59169.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59169.log.gz 2024-09-16T23:43:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48443.log 2024-09-16T23:43:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.41294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42603.log 2024-09-16T23:43:17.072 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48443.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48443.log.gz 2024-09-16T23:43:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74542.log 2024-09-16T23:43:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42603.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42603.log.gz 2024-09-16T23:43:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48721.log 87.6% 2024-09-16T23:43:17.073 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.41294.log.gz 2024-09-16T23:43:17.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74542.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74542.log.gz 2024-09-16T23:43:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64455.log 2024-09-16T23:43:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48721.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48721.log.gz 2024-09-16T23:43:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70002.log 2024-09-16T23:43:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64455.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64455.log.gz 2024-09-16T23:43:17.074 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52249.log 2024-09-16T23:43:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70002.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70002.log.gz 2024-09-16T23:43:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67292.log 2024-09-16T23:43:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52249.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52249.log.gz 2024-09-16T23:43:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73510.log 2024-09-16T23:43:17.075 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67292.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67292.log.gz 2024-09-16T23:43:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70789.log 2024-09-16T23:43:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73510.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.73510.log.gz 2024-09-16T23:43:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62973.log 2024-09-16T23:43:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70789.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.70789.log.gz 2024-09-16T23:43:17.076 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40786.log 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62973.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62973.log.gz 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60083.log 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.3.40786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73114.log 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60083.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60083.log.gz 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73156.log 2024-09-16T23:43:17.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73114.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73114.log.gz 2024-09-16T23:43:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69463.log 2024-09-16T23:43:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73156.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73156.log.gz 2024-09-16T23:43:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39230.log 2024-09-16T23:43:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69463.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69463.log.gz 2024-09-16T23:43:17.078 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41218.log 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70293.log 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39230.log.gz 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47940.log 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41218.log.gz 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70293.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.70293.log.gz 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40786.log.gz 2024-09-16T23:43:17.079 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69834.log 2024-09-16T23:43:17.080 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47940.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47940.log.gz 2024-09-16T23:43:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50847.log 2024-09-16T23:43:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69834.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73461.log.gz 2024-09-16T23:43:17.083 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44983.log 2024-09-16T23:43:17.084 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50847.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50847.log.gz 2024-09-16T23:43:17.085 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69834.log.gz 2024-09-16T23:43:17.091 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46682.log 2024-09-16T23:43:17.093 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44983.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44983.log.gz 2024-09-16T23:43:17.103 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35816.log 2024-09-16T23:43:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46682.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46682.log.gz 2024-09-16T23:43:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70834.log 2024-09-16T23:43:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35816.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35816.log.gz 2024-09-16T23:43:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68484.log 2024-09-16T23:43:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70834.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.70834.log.gz 2024-09-16T23:43:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64827.log 2024-09-16T23:43:17.116 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68484.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46106.log.gz 2024-09-16T23:43:17.117 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38035.log 2024-09-16T23:43:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64827.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64827.log.gz 2024-09-16T23:43:17.126 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74668.log 2024-09-16T23:43:17.129 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70943.log 2024-09-16T23:43:17.129 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74668.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74668.log.gz 2024-09-16T23:43:17.129 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66640.log 2024-09-16T23:43:17.129 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70943.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70943.log.gz 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54114.log 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70811.log.gz 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66640.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66640.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64124.log 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54114.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54114.log.gz 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50333.log 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64124.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64124.log.gz 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr: 63.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38035.log.gz 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60763.log 2024-09-16T23:43:17.133 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50333.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50333.log.gz 2024-09-16T23:43:17.134 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73719.log 2024-09-16T23:43:17.135 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67450.log 2024-09-16T23:43:17.135 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73719.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73719.log.gz 2024-09-16T23:43:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55891.log 2024-09-16T23:43:17.136 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60763.log.gz 2024-09-16T23:43:17.136 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67450.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67450.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67540.log 2024-09-16T23:43:17.137 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55891.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55891.log.gz 2024-09-16T23:43:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56156.log 2024-09-16T23:43:17.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64558.log 2024-09-16T23:43:17.139 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67540.log.gz 2024-09-16T23:43:17.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56156.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56156.log.gz 2024-09-16T23:43:17.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62523.log 2024-09-16T23:43:17.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64558.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64558.log.gz 2024-09-16T23:43:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43587.log 2024-09-16T23:43:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62523.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62523.log.gz 2024-09-16T23:43:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66550.log 2024-09-16T23:43:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43587.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43587.log.gz 2024-09-16T23:43:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50516.log 2024-09-16T23:43:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66550.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66550.log.gz 2024-09-16T23:43:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50516.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50516.log.gz 2024-09-16T23:43:17.148 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59394.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51140.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60223.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51140.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59394.log.gz 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51140.log.gz 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49655.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67022.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60223.log.gz 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61374.log 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49655.log.gz 2024-09-16T23:43:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61736.log 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61374.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61374.log.gz 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67607.log 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61736.log: /var/log/ceph/cluster1-client.mirror.67022.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67022.log.gz 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68141.log 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67607.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.61736.log.gz 2024-09-16T23:43:17.152 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67607.log.gz 2024-09-16T23:43:17.154 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68141.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68141.log.gz 2024-09-16T23:43:17.154 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42625.log 2024-09-16T23:43:17.154 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43786.log 2024-09-16T23:43:17.159 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35765.log 2024-09-16T23:43:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43786.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43786.log.gz 2024-09-16T23:43:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58386.log 2024-09-16T23:43:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44623.log 2024-09-16T23:43:17.179 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58386.log: 87.7% 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.35765.log.gz 2024-09-16T23:43:17.179 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58386.log.gz 2024-09-16T23:43:17.180 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51450.log 2024-09-16T23:43:17.180 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44623.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44623.log.gz 2024-09-16T23:43:17.181 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67765.log 2024-09-16T23:43:17.182 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51450.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47999.log.gz 2024-09-16T23:43:17.185 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51450.log.gz 2024-09-16T23:43:17.186 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46724.log 2024-09-16T23:43:17.186 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67765.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.69233.log 2024-09-16T23:43:17.186 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67765.log.gz 2024-09-16T23:43:17.187 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46724.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46724.log.gz 2024-09-16T23:43:17.187 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61422.log 2024-09-16T23:43:17.187 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63791.log 2024-09-16T23:43:17.188 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.2.69233.log: /var/log/ceph/cluster1-client.mirror.61422.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61422.log.gz 2024-09-16T23:43:17.191 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.69233.log.gz 2024-09-16T23:43:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53917.log 2024-09-16T23:43:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63791.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63791.log.gz 2024-09-16T23:43:17.193 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60127.log 2024-09-16T23:43:17.193 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64765.log 2024-09-16T23:43:17.194 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53917.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53917.log.gz 2024-09-16T23:43:17.194 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71609.log 2024-09-16T23:43:17.195 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64765.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64765.log.gz 2024-09-16T23:43:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55738.log 2024-09-16T23:43:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71609.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71609.log.gz 2024-09-16T23:43:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.48194.log 2024-09-16T23:43:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55738.log: /var/log/ceph/cluster1-client.mirror.60127.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55738.log.gz 2024-09-16T23:43:17.200 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.60127.log.gz 2024-09-16T23:43:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.59969.log 2024-09-16T23:43:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68655.log 2024-09-16T23:43:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47291.log 2024-09-16T23:43:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70670.log 2024-09-16T23:43:17.203 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.2.48194.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64145.log 2024-09-16T23:43:17.203 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70670.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70670.log.gz 2024-09-16T23:43:17.205 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68655.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68655.log.gz 2024-09-16T23:43:17.207 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66257.log 2024-09-16T23:43:17.211 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64145.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64145.log.gz 2024-09-16T23:43:17.213 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73801.log 2024-09-16T23:43:17.213 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66257.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66257.log.gz 2024-09-16T23:43:17.219 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.59969.log: /var/log/ceph/cluster2-client.mirror.47291.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47291.log.gz 2024-09-16T23:43:17.221 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54873.log 2024-09-16T23:43:17.231 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.59969.log.gz 2024-09-16T23:43:17.232 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.42139.log 2024-09-16T23:43:17.232 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54873.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54873.log.gz 2024-09-16T23:43:17.234 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73801.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73801.log.gz 2024-09-16T23:43:17.239 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.55627.log 2024-09-16T23:43:17.249 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.42139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38001.log 2024-09-16T23:43:17.255 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.3.55627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49618.log 2024-09-16T23:43:17.263 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49964.log 2024-09-16T23:43:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49618.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.42139.log.gz 2024-09-16T23:43:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56204.log 2024-09-16T23:43:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.49964.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49618.log.gz 2024-09-16T23:43:17.266 INFO:teuthology.orchestra.run.smithi098.stderr: 78.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49964.log.gz 2024-09-16T23:43:17.266 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65386.log 2024-09-16T23:43:17.267 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56204.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56204.log.gz 2024-09-16T23:43:17.268 INFO:teuthology.orchestra.run.smithi098.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38001.log.gz 2024-09-16T23:43:17.270 INFO:teuthology.orchestra.run.smithi098.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.48194.log.gz 2024-09-16T23:43:17.272 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65386.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65386.log.gz 2024-09-16T23:43:17.273 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69093.log 2024-09-16T23:43:17.281 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39646.log 2024-09-16T23:43:17.281 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69093.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69093.log.gz 2024-09-16T23:43:17.282 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53986.log 2024-09-16T23:43:17.283 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68986.log 2024-09-16T23:43:17.287 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53986.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39646.log.gz 2024-09-16T23:43:17.293 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53986.log.gz 2024-09-16T23:43:17.293 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49273.log 2024-09-16T23:43:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68986.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.68986.log.gz 2024-09-16T23:43:17.295 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40392.log 2024-09-16T23:43:17.304 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71418.log 2024-09-16T23:43:17.310 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.2.40392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56742.log 2024-09-16T23:43:17.317 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.71418.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67967.log 2024-09-16T23:43:17.318 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.56742.log: -- replaced with /var/log/ceph/cluster1-client.mirror.71418.log.gz 2024-09-16T23:43:17.318 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56742.log.gz 2024-09-16T23:43:17.324 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49677.log.gz 2024-09-16T23:43:17.325 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70769.log 2024-09-16T23:43:17.326 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67967.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67967.log.gz 2024-09-16T23:43:17.326 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56031.log 2024-09-16T23:43:17.328 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70769.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70769.log.gz 2024-09-16T23:43:17.328 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62683.log 2024-09-16T23:43:17.328 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56031.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48664.log.gz 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56031.log.gz 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36866.log 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64372.log 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36866.log: /var/log/ceph/cluster2-client.mirror.62683.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36866.log.gz 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62683.log.gz 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44643.log 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61623.log 2024-09-16T23:43:17.334 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64372.log.gz 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73678.log 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61623.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44643.log.gz 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61623.log.gz 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70649.log 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.73678.log: -- /var/log/ceph/cluster2-client.mirror.68860.log 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73678.log.gz 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53451.log 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68860.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.70649.log.gz 2024-09-16T23:43:17.335 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49352.log 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53451.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53451.log.gz 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68860.log.gz 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58896.log 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59214.log 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58896.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58896.log.gz 2024-09-16T23:43:17.338 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71630.log 2024-09-16T23:43:17.339 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59214.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59214.log.gz 2024-09-16T23:43:17.339 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39491.log 2024-09-16T23:43:17.340 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71630.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71630.log.gz 2024-09-16T23:43:17.341 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43193.log 2024-09-16T23:43:17.341 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39491.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47737.log 2024-09-16T23:43:17.342 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43193.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43193.log.gz 2024-09-16T23:43:17.343 INFO:teuthology.orchestra.run.smithi098.stderr: 81.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39491.log.gz 2024-09-16T23:43:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.45227.log 2024-09-16T23:43:17.344 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47737.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47737.log.gz 2024-09-16T23:43:17.345 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44663.log 2024-09-16T23:43:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.45227.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60923.log 2024-09-16T23:43:17.361 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44663.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.44663.log.gz 2024-09-16T23:43:17.367 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47390.log 2024-09-16T23:43:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54721.log 2024-09-16T23:43:17.380 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60923.log.gz 2024-09-16T23:43:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47390.log.gz 2024-09-16T23:43:17.388 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68484.log.gz 2024-09-16T23:43:17.391 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.38365.log.gz 2024-09-16T23:43:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40017.log 2024-09-16T23:43:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46292.log 2024-09-16T23:43:17.394 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72997.log 2024-09-16T23:43:17.395 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46292.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54721.log.gz 2024-09-16T23:43:17.396 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49513.log.gz 2024-09-16T23:43:17.396 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56939.log 2024-09-16T23:43:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72997.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40017.log.gz 2024-09-16T23:43:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54070.log 2024-09-16T23:43:17.401 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56939.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56939.log.gz 2024-09-16T23:43:17.402 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59193.log 2024-09-16T23:43:17.402 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54070.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54070.log.gz 2024-09-16T23:43:17.403 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57920.log 2024-09-16T23:43:17.403 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59193.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59193.log.gz 2024-09-16T23:43:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.45227.log.gz 2024-09-16T23:43:17.405 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.44963.log 2024-09-16T23:43:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57920.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72162.log 2024-09-16T23:43:17.405 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57920.log.gz 2024-09-16T23:43:17.405 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44963.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44963.log.gz 2024-09-16T23:43:17.407 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66730.log 2024-09-16T23:43:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57429.log 2024-09-16T23:43:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72162.log: /var/log/ceph/cluster1-client.mirror.66730.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72162.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66730.log.gz 2024-09-16T23:43:17.408 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65241.log 2024-09-16T23:43:17.408 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41912.log 2024-09-16T23:43:17.409 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57429.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57429.log.gz 2024-09-16T23:43:17.409 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51044.log 2024-09-16T23:43:17.411 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41912.log: /var/log/ceph/cluster2-client.mirror.65241.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65241.log.gz 2024-09-16T23:43:17.414 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41912.log.gz 2024-09-16T23:43:17.414 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54476.log 2024-09-16T23:43:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51044.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51044.log.gz 2024-09-16T23:43:17.415 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58407.log 2024-09-16T23:43:17.416 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73740.log 2024-09-16T23:43:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54476.log: /var/log/ceph/cluster2-client.mirror.58407.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54476.log.gz 2024-09-16T23:43:17.418 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49066.log 2024-09-16T23:43:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73740.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73740.log.gz 2024-09-16T23:43:17.419 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51514.log 2024-09-16T23:43:17.421 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49066.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49066.log.gz 2024-09-16T23:43:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.58407.log.gz 2024-09-16T23:43:17.422 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.54273.log 2024-09-16T23:43:17.422 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40786.log 2024-09-16T23:43:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51514.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51514.log.gz 2024-09-16T23:43:17.423 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60947.log 2024-09-16T23:43:17.424 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74016.log 2024-09-16T23:43:17.424 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60947.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60947.log.gz 2024-09-16T23:43:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45369.log 2024-09-16T23:43:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.74016.log: /var/log/ceph/cluster2-client.mirror.54273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74016.log.gz 2024-09-16T23:43:17.426 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45450.log 2024-09-16T23:43:17.426 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54273.log.gz 2024-09-16T23:43:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45369.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45369.log.gz 2024-09-16T23:43:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51558.log 2024-09-16T23:43:17.427 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51558.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51558.log.gz 2024-09-16T23:43:17.432 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45450.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45450.log.gz 2024-09-16T23:43:17.432 INFO:teuthology.orchestra.run.smithi098.stderr: 81.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40786.log.gz 2024-09-16T23:43:17.432 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58069.log 2024-09-16T23:43:17.432 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69405.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58069.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58069.log.gz 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.55627.log.gz 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44259.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69405.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58362.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44259.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69405.log.gz 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54652.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58362.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58362.log.gz 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46491.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54652.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54652.log.gz 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-16T23:43:17.436 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46491.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46491.log.gz 2024-09-16T23:43:17.437 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59145.log 2024-09-16T23:43:17.437 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65758.log 2024-09-16T23:43:17.437 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-osd.0.log: /var/log/ceph/cluster1-client.mirror.59145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50494.log 2024-09-16T23:43:17.439 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59145.log.gz 2024-09-16T23:43:17.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65758.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44259.log.gz 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65758.log.gz 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36110.log 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.50494.log: -- /var/log/ceph/cluster1-client.mirror.41819.log 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36110.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36110.log.gz 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36953.log 2024-09-16T23:43:17.440 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41819.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50494.log.gz 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41819.log.gz 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61470.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.36953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60899.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55983.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61470.log.gz/var/log/ceph/cluster1-client.mirror.60899.log: 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49454.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.60899.log.gz 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55983.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55983.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44943.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57074.log 2024-09-16T23:43:17.453 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44943.log: /var/log/ceph/cluster2-client.mirror.49454.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49454.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr: 73.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45841.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44943.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36953.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71650.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57074.log: /var/log/ceph/cluster2-client.mirror.45841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39401.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57074.log.gz 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45841.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71650.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71650.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66572.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41630.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39401.log: /var/log/ceph/cluster1-client.mirror.66572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66685.log 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41630.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66572.log.gz 2024-09-16T23:43:17.454 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39278.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53783.log 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66685.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46233.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53783.log: /var/log/ceph/cluster1-client.mirror.39278.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53783.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68778.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56094.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68778.log: /var/log/ceph/cluster2-client.mirror.46233.log: 76.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46233.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68778.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60649.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54741.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60649.log: /var/log/ceph/cluster2-client.mirror.56094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60649.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56094.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45190.log 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54741.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39278.log.gz 2024-09-16T23:43:17.455 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53580.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45190.log: 92.4% 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45190.log.gz 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.54741.log.gz 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42518.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48581.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42518.log: /var/log/ceph/cluster2-client.mirror.53580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46645.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53580.log.gz 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58186.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46645.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46645.log.gz 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67922.log 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58186.log: /var/log/ceph/cluster2-client.mirror.48581.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58186.log.gz 2024-09-16T23:43:17.464 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58427.log 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42518.log.gz 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70588.log 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58427.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67922.log.gz 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39783.log 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70588.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70588.log.gz 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58427.log.gz 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43230.log 2024-09-16T23:43:17.465 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44345.log 2024-09-16T23:43:17.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58852.log 2024-09-16T23:43:17.491 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58565.log 2024-09-16T23:43:17.492 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58852.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58852.log.gz 2024-09-16T23:43:17.495 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44345.log.gz 2024-09-16T23:43:17.496 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59706.log 2024-09-16T23:43:17.500 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58565.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58565.log.gz 2024-09-16T23:43:17.500 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56052.log 2024-09-16T23:43:17.501 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59706.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59706.log.gz 2024-09-16T23:43:17.504 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50823.log 2024-09-16T23:43:17.506 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56052.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56052.log.gz 2024-09-16T23:43:17.506 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56787.log 2024-09-16T23:43:17.508 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50823.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39783.log.gz 2024-09-16T23:43:17.508 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50823.log.gz 2024-09-16T23:43:17.509 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55584.log 2024-09-16T23:43:17.513 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56787.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56787.log.gz 2024-09-16T23:43:17.517 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62870.log 2024-09-16T23:43:17.517 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55584.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55584.log.gz 2024-09-16T23:43:17.518 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57762.log 2024-09-16T23:43:17.519 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62870.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62870.log.gz 2024-09-16T23:43:17.519 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69446.log 2024-09-16T23:43:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57762.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57762.log.gz 2024-09-16T23:43:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63472.log 2024-09-16T23:43:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69446.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69446.log.gz 2024-09-16T23:43:17.534 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72670.log 2024-09-16T23:43:17.535 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63472.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63472.log.gz 2024-09-16T23:43:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72079.log 2024-09-16T23:43:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72670.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72670.log.gz 2024-09-16T23:43:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44537.log 2024-09-16T23:43:17.539 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43230.log.gz 2024-09-16T23:43:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55411.log 2024-09-16T23:43:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44537.log: /var/log/ceph/cluster1-client.mirror.72079.log: 86.7% 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44537.log.gz 2024-09-16T23:43:17.544 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72079.log.gz 2024-09-16T23:43:17.544 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71879.log 2024-09-16T23:43:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68407.log 2024-09-16T23:43:17.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55411.log: /var/log/ceph/cluster2-client.mirror.71879.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55411.log.gz 2024-09-16T23:43:17.549 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39401.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64041.log 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3%/var/log/ceph/cluster2-client.mirror.68407.log: -- replaced with /var/log/ceph/cluster2-client.mirror.71879.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr: 42.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68407.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37574.log 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60199.log 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37574.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45885.log 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64041.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58743.log 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45885.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60199.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45885.log.gz 2024-09-16T23:43:17.550 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68009.log 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58743.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58743.log.gz 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54805.log 63.6% 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.37574.log.gz 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68009.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68009.log.gz 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52855.log 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72473.log 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52855.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52855.log.gz 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54805.log.gz 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73197.log 2024-09-16T23:43:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72473.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61080.log 2024-09-16T23:43:17.557 INFO:teuthology.orchestra.run.smithi098.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72473.log.gz 2024-09-16T23:43:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73197.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73197.log.gz 2024-09-16T23:43:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.49964.log 2024-09-16T23:43:17.558 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61080.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61080.log.gz 2024-09-16T23:43:17.558 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65592.log 2024-09-16T23:43:17.559 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.1.49964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60286.log 2024-09-16T23:43:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65592.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.49964.log.gz 2024-09-16T23:43:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63640.log 2024-09-16T23:43:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60286.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60286.log.gz 2024-09-16T23:43:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36296.log 2024-09-16T23:43:17.568 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65592.log.gz/var/log/ceph/cluster2-client.mirror.63640.log: 2024-09-16T23:43:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63640.log.gz 2024-09-16T23:43:17.568 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.59664.log 2024-09-16T23:43:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52389.log 2024-09-16T23:43:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36296.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36296.log.gz 2024-09-16T23:43:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59664.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59664.log.gz 2024-09-16T23:43:17.570 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61716.log 2024-09-16T23:43:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52389.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52389.log.gz 2024-09-16T23:43:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65902.log 2024-09-16T23:43:17.571 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69323.log 2024-09-16T23:43:17.573 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65902.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61716.log.gz 2024-09-16T23:43:17.574 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69323.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69323.log.gz 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65902.log.gz 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59031.log 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72955.log 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65344.log 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72955.log: /var/log/ceph/cluster2-client.mirror.59031.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59031.log.gz 2024-09-16T23:43:17.577 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55827.log 2024-09-16T23:43:17.578 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65344.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65344.log.gz 2024-09-16T23:43:17.579 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43915.log 2024-09-16T23:43:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55827.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72955.log.gz 2024-09-16T23:43:17.580 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41196.log 2024-09-16T23:43:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43915.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55827.log.gz 2024-09-16T23:43:17.582 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37523.log 2024-09-16T23:43:17.583 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41196.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68898.log 2024-09-16T23:43:17.584 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65510.log 2024-09-16T23:43:17.584 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42625.log.gz 2024-09-16T23:43:17.585 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68898.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68898.log.gz 2024-09-16T23:43:17.585 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63051.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65510.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65510.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41478.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43915.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.63051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55807.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41478.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63051.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37523.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr: /var/log/ceph/cluster1-client.mirror.44150.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43385.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66122.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43385.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43385.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55807.log.gz 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70731.log 2024-09-16T23:43:17.591 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53335.log 2024-09-16T23:43:17.592 INFO:teuthology.orchestra.run.smithi098.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66122.log.gz 2024-09-16T23:43:17.592 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53335.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53335.log.gz 2024-09-16T23:43:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70731.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.70731.log.gz 2024-09-16T23:43:17.606 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44150.log.gz 2024-09-16T23:43:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52697.log 2024-09-16T23:43:17.606 INFO:teuthology.orchestra.run.smithi098.stderr:gzip 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41478.log.gz 2024-09-16T23:43:17.606 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68321.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65613.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52697.log.gz 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41406.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65613.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68321.log.gz 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65613.log.gz 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47957.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65716.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47957.log: /var/log/ceph/cluster1-client.mirror.41406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39067.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65716.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47957.log.gz 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39622.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52293.log 2024-09-16T23:43:17.607 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39067.log: /var/log/ceph/cluster2-client.mirror.39622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71771.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46571.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65716.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52293.log: /var/log/ceph/cluster2-client.mirror.71771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67585.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46571.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52293.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71771.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46571.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67585.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71900.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67585.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39901.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65138.log 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41406.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39901.log: /var/log/ceph/cluster2-client.mirror.71900.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.71900.log.gz 2024-09-16T23:43:17.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65138.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39067.log.gz 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59302.log 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65138.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65179.log 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66707.log 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59302.log: /var/log/ceph/cluster2-client.mirror.65179.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63873.log 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65179.log.gz 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66707.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66707.log.gz 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59302.log.gz 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52430.log 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39622.log.gz 2024-09-16T23:43:17.609 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63873.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70145.log 2024-09-16T23:43:17.614 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52430.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63873.log.gz 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.39901.log.gz 2024-09-16T23:43:17.614 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70629.log 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66617.log 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70629.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52430.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45535.log 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70629.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66617.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66617.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48782.log 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.70145.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68231.log 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48782.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45535.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr: 82.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48782.log.gz 2024-09-16T23:43:17.615 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47841.log 2024-09-16T23:43:17.616 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68231.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68231.log.gz 2024-09-16T23:43:17.616 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64413.log 2024-09-16T23:43:17.616 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45004.log 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57699.log 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45004.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64413.log.gz 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45004.log.gz 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49373.log 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65654.log 2024-09-16T23:43:17.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57699.log: /var/log/ceph/cluster2-client.mirror.49373.log: 79.5% 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57699.log.gz 2024-09-16T23:43:17.619 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49373.log.gz 2024-09-16T23:43:17.619 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45982.log 2024-09-16T23:43:17.619 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46270.log 2024-09-16T23:43:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65654.log: /var/log/ceph/cluster2-client.mirror.45982.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45982.log.gz 2024-09-16T23:43:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72490.log 2024-09-16T23:43:17.621 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46270.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65654.log.gz 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40137.log 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46270.log.gz 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72490.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72490.log.gz 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41196.log.gz 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45724.log 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6%/var/log/ceph/cluster1-client.mirror.40137.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46292.log.gz 2024-09-16T23:43:17.622 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49046.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58989.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49046.log: /var/log/ceph/cluster2-client.mirror.45724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50357.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45724.log.gz 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58989.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49046.log.gz 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56180.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51385.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.56180.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50357.log.gz 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52207.log 2024-09-16T23:43:17.630 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51385.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51385.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58989.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56180.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40137.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39023.log 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41267.log 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4%/var/log/ceph/cluster1-client.admin.39023.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52207.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56297.log 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53379.log 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37105.log 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53379.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53379.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56297.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.39023.log.gz 2024-09-16T23:43:17.631 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.41267.log.gz 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51409.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63410.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51409.log: gzip/var/log/ceph/cluster2-client.mirror.63410.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63616.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-mon.a.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63410.log.gz 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51409.log.gz 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47470.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63616.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49231.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45741.log 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.49231.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63616.log.gz 2024-09-16T23:43:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65696.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45741.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49231.log.gz 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51116.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45741.log.gz/var/log/ceph/cluster2-client.mirror.65696.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65696.log.gz 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67225.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47470.log.gz 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43935.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51116.log: /var/log/ceph/cluster1-client.mirror.67225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71688.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67225.log.gz 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51116.log.gz 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50164.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55560.log 2024-09-16T23:43:17.657 INFO:teuthology.orchestra.run.smithi098.stderr: 85.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71688.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50164.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36824.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55560.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50164.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55190.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55560.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36824.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72244.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55190.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36824.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65220.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55190.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55456.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65220.log: /var/log/ceph/cluster1-client.mirror.43935.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72244.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43119.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55456.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65220.log.gz 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58695.log 2024-09-16T23:43:17.658 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55456.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57381.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58695.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58695.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47782.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57381.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57381.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69072.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37105.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47782.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47782.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43935.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71306.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43119.log: /var/log/ceph/cluster2-client.mirror.69072.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69072.log.gz 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64868.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38145.log 2024-09-16T23:43:17.659 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.71306.log: /var/log/ceph/cluster2-client.mirror.64868.log: 85.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71306.log.gz 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64868.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47087.log 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58606.log 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38145.log: /var/log/ceph/cluster2-client.mirror.47087.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47087.log.gz 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56425.log 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63182.log 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56425.log: /var/log/ceph/cluster2-client.mirror.58606.log: 91.3% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58606.log.gz 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56425.log.gz 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51940.log 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63182.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63182.log.gz 2024-09-16T23:43:17.660 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54383.log 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51940.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38145.log.gz 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51940.log.gz 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43649.log 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54383.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54383.log.gz 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52317.log 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43649.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43649.log.gz 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49806.log 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52317.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52317.log.gz 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47678.log 2024-09-16T23:43:17.666 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49806.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49806.log.gz 2024-09-16T23:43:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42851.log 2024-09-16T23:43:17.667 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47678.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69483.log 2024-09-16T23:43:17.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42851.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42851.log.gz 2024-09-16T23:43:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71568.log 2024-09-16T23:43:17.669 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69483.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.69483.log.gz 2024-09-16T23:43:17.671 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71743.log 2024-09-16T23:43:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71568.log: 86.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71568.log.gz 2024-09-16T23:43:17.676 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45594.log 2024-09-16T23:43:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71743.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71743.log.gz 2024-09-16T23:43:17.677 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42650.log 2024-09-16T23:43:17.686 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45594.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49352.log.gz 2024-09-16T23:43:17.688 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63894.log 2024-09-16T23:43:17.689 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.42650.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.42650.log.gz 2024-09-16T23:43:17.698 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73279.log 2024-09-16T23:43:17.702 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63894.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63894.log.gz 2024-09-16T23:43:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47700.log 2024-09-16T23:43:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73279.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73279.log.gz 2024-09-16T23:43:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55295.log 2024-09-16T23:43:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47700.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47700.log.gz 2024-09-16T23:43:17.733 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61599.log 2024-09-16T23:43:17.734 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55295.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55295.log.gz 2024-09-16T23:43:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45651.log 2024-09-16T23:43:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61599.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.61599.log.gz 2024-09-16T23:43:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51472.log 2024-09-16T23:43:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45651.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45651.log.gz 2024-09-16T23:43:17.751 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55762.log 2024-09-16T23:43:17.758 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51472.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.51472.log.gz 2024-09-16T23:43:17.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54455.log 2024-09-16T23:43:17.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55762.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55762.log.gz 2024-09-16T23:43:17.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46002.log 2024-09-16T23:43:17.767 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54455.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54455.log.gz 2024-09-16T23:43:17.770 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38241.log 2024-09-16T23:43:17.771 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46002.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46002.log.gz 2024-09-16T23:43:17.772 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42991.log 2024-09-16T23:43:17.773 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.38241.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49273.log.gz 2024-09-16T23:43:17.773 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.38241.log 2024-09-16T23:43:17.774 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42991.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42991.log.gz 2024-09-16T23:43:17.774 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72532.log 2024-09-16T23:43:17.775 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.38241.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39182.log 2024-09-16T23:43:17.777 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72532.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.38241.log.gz 2024-09-16T23:43:17.780 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63141.log 2024-09-16T23:43:17.788 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63306.log 2024-09-16T23:43:17.789 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63141.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63141.log.gz 2024-09-16T23:43:17.791 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39182.log.gz 2024-09-16T23:43:17.792 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55008.log 2024-09-16T23:43:17.793 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63306.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63306.log.gz 2024-09-16T23:43:17.794 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48701.log 2024-09-16T23:43:17.794 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55008.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55008.log.gz 2024-09-16T23:43:17.796 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52807.log 2024-09-16T23:43:17.797 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48701.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48701.log.gz 2024-09-16T23:43:17.798 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74584.log 2024-09-16T23:43:17.799 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52807.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52807.log.gz 2024-09-16T23:43:17.799 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37407.log 2024-09-16T23:43:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74584.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74584.log.gz 2024-09-16T23:43:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49251.log 2024-09-16T23:43:17.800 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.37407.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.37407.log.gz 2024-09-16T23:43:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66482.log 2024-09-16T23:43:17.802 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49251.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49251.log.gz 2024-09-16T23:43:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-16T23:43:17.803 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66482.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66482.log.gz 2024-09-16T23:43:17.804 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66325.log 2024-09-16T23:43:17.804 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64393.log 2024-09-16T23:43:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66325.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66325.log.gz 2024-09-16T23:43:17.806 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73361.log 2024-09-16T23:43:17.807 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64393.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64393.log.gz 2024-09-16T23:43:17.807 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48098.log 2024-09-16T23:43:17.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73361.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.73361.log.gz 2024-09-16T23:43:17.809 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37540.log 2024-09-16T23:43:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48098.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48098.log.gz 2024-09-16T23:43:17.811 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63327.log 2024-09-16T23:43:17.812 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41016.log 2024-09-16T23:43:17.813 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63327.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63327.log.gz 2024-09-16T23:43:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64971.log 2024-09-16T23:43:17.814 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41016.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.37540.log.gz 2024-09-16T23:43:17.816 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56626.log 2024-09-16T23:43:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64971.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64971.log.gz 2024-09-16T23:43:17.818 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49789.log 2024-09-16T23:43:17.819 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56626.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56626.log.gz 2024-09-16T23:43:17.819 INFO:teuthology.orchestra.run.smithi098.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41016.log.gz 2024-09-16T23:43:17.826 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41240.log 2024-09-16T23:43:17.827 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49789.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49789.log.gz 2024-09-16T23:43:17.828 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56807.log 2024-09-16T23:43:17.831 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63492.log 2024-09-16T23:43:17.837 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56807.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56807.log.gz 2024-09-16T23:43:17.837 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38069.log 2024-09-16T23:43:17.838 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63492.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63492.log.gz 2024-09-16T23:43:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64269.log 2024-09-16T23:43:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.36978.log 2024-09-16T23:43:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64269.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64269.log.gz 2024-09-16T23:43:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35900.log 2024-09-16T23:43:17.850 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.1.36978.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38069.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42967.log 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35900.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35900.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62890.log 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42967.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72997.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42967.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72711.log 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47878.log 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72711.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62890.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.48782.log 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.72711.log.gz 2024-09-16T23:43:17.851 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51675.log 2024-09-16T23:43:17.852 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47878.log.gz 2024-09-16T23:43:17.853 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72121.log 2024-09-16T23:43:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51675.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51675.log.gz 2024-09-16T23:43:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 75.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.36978.log.gz 2024-09-16T23:43:17.855 INFO:teuthology.orchestra.run.smithi098.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.66032.log 2024-09-16T23:43:17.855 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72121.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72121.log.gz 2024-09-16T23:43:17.856 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.mirror.3.48782.log: -- /var/log/ceph/cluster2-client.mirror.43829.log 2024-09-16T23:43:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66032.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66032.log.gz 2024-09-16T23:43:17.857 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51337.log 2024-09-16T23:43:17.858 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43829.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.48782.log.gz 2024-09-16T23:43:17.859 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35740.log 2024-09-16T23:43:17.859 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51337.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51337.log.gz 2024-09-16T23:43:17.859 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72056.log 2024-09-16T23:43:17.864 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43829.log.gz 2024-09-16T23:43:17.864 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59527.log 2024-09-16T23:43:17.864 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72056.log: /var/log/ceph/cluster2-client.admin.35740.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72056.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35740.log.gz 2024-09-16T23:43:17.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.865 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50453.log 2024-09-16T23:43:17.865 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56584.log 2024-09-16T23:43:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50453.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59527.log.gz 2024-09-16T23:43:17.868 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50453.log.gz 2024-09-16T23:43:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62049.log 2024-09-16T23:43:17.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56584.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56584.log.gz 2024-09-16T23:43:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59619.log 2024-09-16T23:43:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62049.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43341.log 2024-09-16T23:43:17.869 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62049.log.gz 2024-09-16T23:43:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59619.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59619.log.gz 2024-09-16T23:43:17.869 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57542.log 2024-09-16T23:43:17.870 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43341.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43341.log.gz 2024-09-16T23:43:17.871 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53538.log 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43673.log 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53538.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57542.log.gz 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67495.log 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43673.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43673.log.gz 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70752.log 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67495.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67495.log.gz 2024-09-16T23:43:17.877 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52739.log 2024-09-16T23:43:17.878 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53538.log.gz 2024-09-16T23:43:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45227.log 2024-09-16T23:43:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52739.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70752.log.gz 2024-09-16T23:43:17.878 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41040.log 2024-09-16T23:43:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.45227.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45227.log.gz 2024-09-16T23:43:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52739.log.gz 2024-09-16T23:43:17.883 INFO:teuthology.orchestra.run.smithi098.stderr: -- /var/log/ceph/cluster1-client.admin.43140.log 2024-09-16T23:43:17.883 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73026.log 2024-09-16T23:43:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.43140.log: /var/log/ceph/cluster1-client.mirror.41040.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53030.log 2024-09-16T23:43:17.884 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.43140.log.gz 2024-09-16T23:43:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73026.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73026.log.gz 2024-09-16T23:43:17.884 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60601.log 2024-09-16T23:43:17.884 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41040.log.gz 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53030.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53030.log.gz 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65551.log 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40113.log 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60601.log.gz 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65551.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65551.log.gz 2024-09-16T23:43:17.886 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52538.log 2024-09-16T23:43:17.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66842.log 2024-09-16T23:43:17.887 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52538.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52538.log.gz 2024-09-16T23:43:17.899 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57944.log 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72897.log 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66842.log: 86.8%/var/log/ceph/cluster1-client.mirror.57944.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66842.log.gz 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62298.log 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.72897.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57944.log.gz 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72897.log.gzgzip 2024-09-16T23:43:17.900 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61124.log 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59256.log 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61124.log: /var/log/ceph/cluster1-client.mirror.62298.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61124.log.gz 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36614.log 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59256.log: -- replaced with /var/log/ceph/cluster1-client.mirror.40113.log.gz 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62298.log.gz 2024-09-16T23:43:17.901 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59256.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70104.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.36614.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36614.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47183.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36488.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70104.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43409.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.36488.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.36488.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50895.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43409.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43409.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66302.log 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47183.log: /var/log/ceph/cluster1-client.mirror.50895.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47183.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50895.log.gz 2024-09-16T23:43:17.902 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40972.log 2024-09-16T23:43:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60625.log 2024-09-16T23:43:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49310.log 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60625.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40972.log.gz 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60625.log.gz 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66302.log.gz 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71589.log 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61848.log 2024-09-16T23:43:17.907 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49310.log.gz 2024-09-16T23:43:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71589.log: gzip/var/log/ceph/cluster1-client.mirror.61848.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61848.log.gz 2024-09-16T23:43:17.908 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71589.log.gz 2024-09-16T23:43:17.908 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46613.log 2024-09-16T23:43:17.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48115.log 2024-09-16T23:43:17.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64992.log 2024-09-16T23:43:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48115.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48115.log.gz 2024-09-16T23:43:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45338.log 2024-09-16T23:43:17.912 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64992.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64992.log.gz 2024-09-16T23:43:17.915 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52048.log 2024-09-16T23:43:17.920 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45338.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43119.log.gz 2024-09-16T23:43:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48539.log 2024-09-16T23:43:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52048.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52048.log.gz 2024-09-16T23:43:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49905.log 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48539.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48539.log.gz 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37455.log 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45594.log.gz 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45094.log 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.37455.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49905.log.gz 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40683.log 2024-09-16T23:43:17.954 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56766.log 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45094.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40683.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62636.log 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56252.log 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.37455.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62636.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56766.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57675.log 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56252.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56252.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44883.log 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57675.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62636.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57675.log.gz 2024-09-16T23:43:17.955 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60513.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.44883.log: --verbose -- /var/log/ceph/cluster2-client.mirror.72938.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44883.log.gz 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40329.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72938.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72938.log.gz 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62208.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.40329.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60513.log.gz 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57651.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62208.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41586.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62208.log.gz 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60697.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41586.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57651.log.gz 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53251.log 2024-09-16T23:43:17.956 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65634.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60697.log.gz 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41133.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65634.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65634.log.gz 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58314.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62097.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58314.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.40329.log.gz 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53251.log.gz 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58314.log.gz 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48036.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43978.log 2024-09-16T23:43:17.957 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62097.log.gz 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48036.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58521.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43978.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48036.log.gz 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57136.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58521.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58521.log.gz 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40392.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57136.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57136.log.gz 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41654.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51205.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43978.log.gz 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61758.log 2024-09-16T23:43:17.958 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.41654.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59969.log 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61758.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40392.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51205.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61758.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41975.log 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59969.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.59969.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65468.log 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41975.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51205.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61262.log 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65468.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65468.log.gz 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61350.log 2024-09-16T23:43:17.959 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61262.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61262.log.gz 2024-09-16T23:43:17.960 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41975.log.gz 2024-09-16T23:43:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46143.log 2024-09-16T23:43:17.960 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61350.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61350.log.gz 2024-09-16T23:43:17.974 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57501.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58093.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.46143.log: /var/log/ceph/cluster1-client.mirror.57501.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57501.log.gz 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37057.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.58093.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58093.log.gz 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71818.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.37057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41707.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71818.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71818.log.gz 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46143.log.gz 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71506.log 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41707.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37057.log.gz 2024-09-16T23:43:17.975 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47898.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41707.log.gz 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55094.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47898.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47898.log.gz 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67067.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55094.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55094.log.gz 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54335.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67067.log: /var/log/ceph/cluster1-client.mirror.71506.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67067.log.gz 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71506.log.gz 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59370.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54335.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54335.log.gz 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36446.log 2024-09-16T23:43:17.976 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59370.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59370.log.gz 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48460.log 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36446.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68593.log 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.36446.log.gz 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48888.log 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48460.log: /var/log/ceph/cluster1-client.mirror.68593.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68593.log.gz 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48283.log 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.48888.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48460.log.gz 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48888.log.gz 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61579.log 2024-09-16T23:43:17.977 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48283.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48283.log.gz 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39454.log 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64021.log 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39454.log: gzip 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61579.log.gz 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73842.log 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58007.log 2024-09-16T23:43:17.978 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64021.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73842.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73842.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46741.log 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58007.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58007.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39454.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73970.log 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.41586.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42400.log 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46741.log: /var/log/ceph/cluster2-client.mirror.73970.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73970.log.gz 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46741.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45509.log 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:17.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52494.log 2024-09-16T23:43:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42400.log: /var/log/ceph/cluster2-client.mirror.45509.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42400.log.gz 2024-09-16T23:43:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45821.log 2024-09-16T23:43:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52494.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52494.log.gz 2024-09-16T23:43:17.985 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62073.log 2024-09-16T23:43:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45821.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45821.log.gz 2024-09-16T23:43:17.988 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69051.log 2024-09-16T23:43:17.990 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62073.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62073.log.gz 2024-09-16T23:43:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45316.log 2024-09-16T23:43:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69051.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69051.log.gz 2024-09-16T23:43:17.991 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47492.log 2024-09-16T23:43:17.994 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45316.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45316.log.gz 2024-09-16T23:43:17.995 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42472.log 2024-09-16T23:43:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44021.log 2024-09-16T23:43:18.024 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42472.log: 89.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43958.log 2024-09-16T23:43:18.024 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.42472.log.gz 2024-09-16T23:43:18.030 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43502.log 2024-09-16T23:43:18.036 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43958.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44021.log.gz 2024-09-16T23:43:18.049 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67202.log 2024-09-16T23:43:18.049 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43502.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43958.log.gz 2024-09-16T23:43:18.053 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48581.log.gz 2024-09-16T23:43:18.056 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48522.log 2024-09-16T23:43:18.056 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67202.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.67202.log.gz 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41891.log 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48522.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48522.log.gz 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37135.log 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41891.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41891.log.gz 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40683.log.gz 2024-09-16T23:43:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63513.log 2024-09-16T23:43:18.060 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71107.log 2024-09-16T23:43:18.060 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63513.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63513.log.gz 86.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37135.log.gz 2024-09-16T23:43:18.061 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71959.log 2024-09-16T23:43:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71107.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.71107.log.gz 2024-09-16T23:43:18.061 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68505.log 2024-09-16T23:43:18.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66145.log 2024-09-16T23:43:18.074 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68505.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68505.log.gz 2024-09-16T23:43:18.080 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52096.log 2024-09-16T23:43:18.088 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52514.log 2024-09-16T23:43:18.090 INFO:teuthology.orchestra.run.smithi098.stderr: 87.0%/var/log/ceph/cluster1-client.mirror.52096.log: -- replaced with /var/log/ceph/cluster1-client.mirror.66145.log.gz 2024-09-16T23:43:18.090 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52096.log.gz 2024-09-16T23:43:18.107 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47841.log.gz 2024-09-16T23:43:18.107 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53845.log 2024-09-16T23:43:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52514.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52514.log.gz 2024-09-16T23:43:18.108 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67157.log 2024-09-16T23:43:18.110 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53845.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53845.log.gz 2024-09-16T23:43:18.110 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66932.log 2024-09-16T23:43:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67157.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67157.log.gz 2024-09-16T23:43:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67742.log 2024-09-16T23:43:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66932.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66932.log.gz 2024-09-16T23:43:18.112 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63389.log 2024-09-16T23:43:18.114 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67742.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67742.log.gz 2024-09-16T23:43:18.114 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62370.log 2024-09-16T23:43:18.114 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63389.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63389.log.gz 2024-09-16T23:43:18.116 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48967.log 2024-09-16T23:43:18.116 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62370.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62370.log.gz 2024-09-16T23:43:18.116 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58449.log 2024-09-16T23:43:18.118 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48967.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48967.log.gz 2024-09-16T23:43:18.118 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42102.log 2024-09-16T23:43:18.119 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58449.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58449.log.gz 2024-09-16T23:43:18.119 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67427.log 2024-09-16T23:43:18.121 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.42102.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42102.log.gz 2024-09-16T23:43:18.121 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65200.log 2024-09-16T23:43:18.122 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67427.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67427.log.gz 2024-09-16T23:43:18.122 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40089.log 2024-09-16T23:43:18.124 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65200.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65200.log.gz 2024-09-16T23:43:18.124 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59643.log 2024-09-16T23:43:18.124 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43763.log 2024-09-16T23:43:18.125 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59643.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59643.log.gz 2024-09-16T23:43:18.126 INFO:teuthology.orchestra.run.smithi098.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40089.log.gz 2024-09-16T23:43:18.127 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48019.log 2024-09-16T23:43:18.132 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43763.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43763.log.gz 2024-09-16T23:43:18.133 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64579.log 2024-09-16T23:43:18.133 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48019.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48019.log.gz 2024-09-16T23:43:18.135 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47333.log 2024-09-16T23:43:18.135 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41430.log 2024-09-16T23:43:18.135 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64579.log.gz 2024-09-16T23:43:18.136 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43502.log.gz 2024-09-16T23:43:18.156 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47333.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41133.log.gz 2024-09-16T23:43:18.162 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50650.log 2024-09-16T23:43:18.166 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41430.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41430.log.gz 2024-09-16T23:43:18.171 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57607.log 2024-09-16T23:43:18.172 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50650.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50650.log.gz 2024-09-16T23:43:18.176 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47678.log.gz 2024-09-16T23:43:18.177 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57607.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57607.log.gz 2024-09-16T23:43:18.181 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57522.log 2024-09-16T23:43:18.182 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71917.log 2024-09-16T23:43:18.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57522.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57522.log.gz 2024-09-16T23:43:18.184 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74708.log 2024-09-16T23:43:18.186 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71917.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71917.log.gz 2024-09-16T23:43:18.186 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49697.log 2024-09-16T23:43:18.186 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74708.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74708.log.gz 2024-09-16T23:43:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46803.log 2024-09-16T23:43:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49697.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49697.log.gz 2024-09-16T23:43:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51430.log 2024-09-16T23:43:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46803.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46803.log.gz 2024-09-16T23:43:18.189 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71991.log 2024-09-16T23:43:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51430.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51430.log.gz 2024-09-16T23:43:18.191 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46704.log 2024-09-16T23:43:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.71991.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.71991.log.gz 2024-09-16T23:43:18.193 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40201.log 2024-09-16T23:43:18.204 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59547.log 2024-09-16T23:43:18.212 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40265.log 2024-09-16T23:43:18.213 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.59547.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59547.log.gz 2024-09-16T23:43:18.220 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46820.log 2024-09-16T23:43:18.224 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70225.log 2024-09-16T23:43:18.225 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46820.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46820.log.gz 2024-09-16T23:43:18.229 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67180.log 2024-09-16T23:43:18.231 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70225.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40201.log.gz 2024-09-16T23:43:18.245 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51068.log 2024-09-16T23:43:18.246 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67180.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67180.log.gz 2024-09-16T23:43:18.259 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70547.log 2024-09-16T23:43:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51068.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51068.log.gz 2024-09-16T23:43:18.273 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52900.log 2024-09-16T23:43:18.274 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70547.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70547.log.gz 2024-09-16T23:43:18.284 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50871.log 2024-09-16T23:43:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52900.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52900.log.gz 2024-09-16T23:43:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61671.log 2024-09-16T23:43:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50871.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50871.log.gz 2024-09-16T23:43:18.308 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51606.log 2024-09-16T23:43:18.309 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61671.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61671.log.gz 2024-09-16T23:43:18.333 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51803.log 2024-09-16T23:43:18.334 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51606.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51606.log.gz 2024-09-16T23:43:18.342 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48135.log 2024-09-16T23:43:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51803.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51803.log.gz 2024-09-16T23:43:18.355 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53271.log 2024-09-16T23:43:18.356 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48135.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48135.log.gz 2024-09-16T23:43:18.364 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.37472.log 2024-09-16T23:43:18.379 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53271.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53271.log.gz 2024-09-16T23:43:18.380 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57627.log 2024-09-16T23:43:18.385 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.37472.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.37472.log.gz 2024-09-16T23:43:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40747.log 2024-09-16T23:43:18.402 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57627.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.57627.log.gz 2024-09-16T23:43:18.414 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38086.log 2024-09-16T23:43:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40747.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46613.log.gz 2024-09-16T23:43:18.424 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45024.log 2024-09-16T23:43:18.428 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.38086.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72532.log.gz 2024-09-16T23:43:18.429 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60581.log 2024-09-16T23:43:18.431 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45024.log: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45024.log.gz 2024-09-16T23:43:18.431 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71128.log 2024-09-16T23:43:18.432 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60581.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.60581.log.gz 2024-09-16T23:43:18.432 INFO:teuthology.orchestra.run.smithi098.stderr: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38086.log.gz 2024-09-16T23:43:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42448.log 2024-09-16T23:43:18.439 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.71128.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71128.log.gz 2024-09-16T23:43:18.446 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72856.log 2024-09-16T23:43:18.447 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42448.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42448.log.gz 2024-09-16T23:43:18.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41088.log 2024-09-16T23:43:18.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72856.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72856.log.gz 2024-09-16T23:43:18.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53134.log 2024-09-16T23:43:18.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66077.log 2024-09-16T23:43:18.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53134.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53134.log.gz 2024-09-16T23:43:18.452 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57052.log 2024-09-16T23:43:18.452 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66077.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66077.log.gz 2024-09-16T23:43:18.452 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43849.log 2024-09-16T23:43:18.456 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57052.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41088.log.gz 2024-09-16T23:43:18.465 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57052.log.gz 2024-09-16T23:43:18.465 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64744.log 2024-09-16T23:43:18.470 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43849.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43849.log.gz 2024-09-16T23:43:18.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73531.log 2024-09-16T23:43:18.472 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47492.log.gz 2024-09-16T23:43:18.472 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64744.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64744.log.gz 2024-09-16T23:43:18.472 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53941.log 2024-09-16T23:43:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73531.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73531.log.gz 2024-09-16T23:43:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49576.log 2024-09-16T23:43:18.474 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43545.log 2024-09-16T23:43:18.476 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49576.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49576.log.gz 2024-09-16T23:43:18.476 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.37057.log 2024-09-16T23:43:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43545.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53941.log.gz 2024-09-16T23:43:18.477 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39319.log 2024-09-16T23:43:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.3.37057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.47183.log 2024-09-16T23:43:18.479 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44236.log 2024-09-16T23:43:18.482 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43545.log.gz 2024-09-16T23:43:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46942.log 2024-09-16T23:43:18.488 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.0.47183.log: /var/log/ceph/cluster1-client.mirror.44236.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44236.log.gz 2024-09-16T23:43:18.488 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46840.log 2024-09-16T23:43:18.503 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40355.log 2024-09-16T23:43:18.503 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46840.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46840.log.gz 2024-09-16T23:43:18.512 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45509.log.gz 2024-09-16T23:43:18.514 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64723.log 2024-09-16T23:43:18.515 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64703.log 2024-09-16T23:43:18.518 INFO:teuthology.orchestra.run.smithi098.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40355.log.gz 2024-09-16T23:43:18.520 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64723.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.47183.log.gz 2024-09-16T23:43:18.520 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48930.log 2024-09-16T23:43:18.520 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64703.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64703.log.gz 2024-09-16T23:43:18.522 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64723.log.gz 2024-09-16T23:43:18.522 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65944.log 2024-09-16T23:43:18.532 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64207.log 2024-09-16T23:43:18.533 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65944.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65944.log.gz 2024-09-16T23:43:18.539 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53182.log 2024-09-16T23:43:18.543 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64207.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64207.log.gz 2024-09-16T23:43:18.543 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50578.log 2024-09-16T23:43:18.544 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53182.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53182.log.gz 2024-09-16T23:43:18.545 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54604.log 2024-09-16T23:43:18.545 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50578.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50578.log.gz 2024-09-16T23:43:18.548 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36953.log 2024-09-16T23:43:18.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54604.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54604.log.gz 2024-09-16T23:43:18.549 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62662.log 2024-09-16T23:43:18.549 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.0.36953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68366.log 2024-09-16T23:43:18.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62662.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62662.log.gz 2024-09-16T23:43:18.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57116.log 2024-09-16T23:43:18.551 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68366.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68366.log.gz 2024-09-16T23:43:18.551 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46761.log 2024-09-16T23:43:18.552 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57116.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57116.log.gz 2024-09-16T23:43:18.552 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48342.log 2024-09-16T23:43:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46761.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46761.log.gz 2024-09-16T23:43:18.553 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67405.log 2024-09-16T23:43:18.566 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48362.log 2024-09-16T23:43:18.567 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67405.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67405.log.gz 2024-09-16T23:43:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56829.log 2024-09-16T23:43:18.578 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48362.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48362.log.gz 2024-09-16T23:43:18.590 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52004.log 2024-09-16T23:43:18.590 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.56829.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56829.log.gz 2024-09-16T23:43:18.596 INFO:teuthology.orchestra.run.smithi098.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.37057.log.gz 2024-09-16T23:43:18.598 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40392.log.gz 2024-09-16T23:43:18.598 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44130.log 2024-09-16T23:43:18.608 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52004.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52004.log.gz 2024-09-16T23:43:18.609 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62499.log 2024-09-16T23:43:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44130.log: 93.5% 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.38241.log.gz 2024-09-16T23:43:18.618 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40619.log 2024-09-16T23:43:18.627 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62499.log: -- replaced with /var/log/ceph/cluster2-client.mirror.44130.log.gz 2024-09-16T23:43:18.629 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62499.log.gz 2024-09-16T23:43:18.629 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45338.log.gz 2024-09-16T23:43:18.631 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.35602.log 2024-09-16T23:43:18.631 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47861.log 2024-09-16T23:43:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.35602.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.35602.log.gz 2024-09-16T23:43:18.632 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60472.log 2024-09-16T23:43:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47861.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47861.log.gz 2024-09-16T23:43:18.633 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39670.log 2024-09-16T23:43:18.634 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60472.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60472.log.gz 2024-09-16T23:43:18.635 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66437.log 2024-09-16T23:43:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52586.log 2024-09-16T23:43:18.636 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66437.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66437.log.gz 2024-09-16T23:43:18.638 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40619.log.gz 2024-09-16T23:43:18.640 INFO:teuthology.orchestra.run.smithi098.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39670.log.gz 2024-09-16T23:43:18.640 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53475.log 2024-09-16T23:43:18.641 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52586.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52586.log.gz 2024-09-16T23:43:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43743.log 2024-09-16T23:43:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53475.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53475.log.gz 2024-09-16T23:43:18.646 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61446.log 2024-09-16T23:43:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.43743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53720.log 2024-09-16T23:43:18.661 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61446.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61446.log.gz 2024-09-16T23:43:18.661 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43743.log.gz 2024-09-16T23:43:18.662 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54676.log 2024-09-16T23:43:18.663 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53720.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53720.log.gz 2024-09-16T23:43:18.664 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-16T23:43:18.664 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54676.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54676.log.gz 2024-09-16T23:43:18.665 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50737.log 2024-09-16T23:43:18.668 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39319.log.gz 2024-09-16T23:43:18.672 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71959.log.gz 2024-09-16T23:43:18.673 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47311.log 2024-09-16T23:43:18.674 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50737.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50737.log.gz 2024-09-16T23:43:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54849.log 2024-09-16T23:43:18.676 INFO:teuthology.orchestra.run.smithi098.stderr: 87.7% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-16T23:43:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69981.log 2024-09-16T23:43:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.54849.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47311.log.gz 2024-09-16T23:43:18.676 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57824.log 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69981.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69981.log.gz 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67720.log 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.57824.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57824.log.gz 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73176.log 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67720.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67720.log.gz 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40725.log 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73176.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54849.log.gz 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73176.log.gz 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68390.log 2024-09-16T23:43:18.684 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56562.log 2024-09-16T23:43:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67090.log 2024-09-16T23:43:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56562.log: /var/log/ceph/cluster1-client.mirror.68390.log: 40.4% -- replaced with /var/log/ceph/cluster1-client.mirror.68390.log.gz 2024-09-16T23:43:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45784.log 2024-09-16T23:43:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67090.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67090.log.gz 2024-09-16T23:43:18.685 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45673.log 2024-09-16T23:43:18.690 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45784.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56562.log.gz 2024-09-16T23:43:18.691 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36149.log 2024-09-16T23:43:18.707 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.37081.log 2024-09-16T23:43:18.707 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36149.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36149.log.gz 2024-09-16T23:43:18.721 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44703.log 2024-09-16T23:43:18.726 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.0.37081.log: 70.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.37081.log.gz 2024-09-16T23:43:18.727 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.70225.log.gz 2024-09-16T23:43:18.729 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65778.log 2024-09-16T23:43:18.743 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.44703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49556.log 2024-09-16T23:43:18.743 INFO:teuthology.orchestra.run.smithi098.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44703.log.gz 2024-09-16T23:43:18.743 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66055.log 2024-09-16T23:43:18.744 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49556.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49556.log.gz 2024-09-16T23:43:18.744 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65778.log.gz 2024-09-16T23:43:18.745 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54315.log 2024-09-16T23:43:18.745 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66055.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66055.log.gz 2024-09-16T23:43:18.746 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58541.log 2024-09-16T23:43:18.747 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54315.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40725.log.gz 2024-09-16T23:43:18.747 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54315.log.gz 2024-09-16T23:43:18.747 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37591.log 2024-09-16T23:43:18.748 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.58541.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58541.log.gz 2024-09-16T23:43:18.748 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40065.log 2024-09-16T23:43:18.749 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.37591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41362.log 2024-09-16T23:43:18.750 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.40065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48157.log 2024-09-16T23:43:18.750 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70922.log 2024-09-16T23:43:18.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48157.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40065.log.gz 2024-09-16T23:43:18.753 INFO:teuthology.orchestra.run.smithi098.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.37591.log.gz 2024-09-16T23:43:18.758 INFO:teuthology.orchestra.run.smithi098.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41362.log.gz 2024-09-16T23:43:18.764 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41997.log 2024-09-16T23:43:18.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70922.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70922.log.gz 2024-09-16T23:43:18.777 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57184.log 2024-09-16T23:43:18.778 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41997.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41997.log.gz 2024-09-16T23:43:18.790 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55070.log 2024-09-16T23:43:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.57184.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48930.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57184.log.gz 2024-09-16T23:43:18.792 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52410.log 2024-09-16T23:43:18.792 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55070.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55070.log.gz 2024-09-16T23:43:18.798 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56987.log 2024-09-16T23:43:18.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.52410.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47333.log.gz 2024-09-16T23:43:18.806 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36023.log 2024-09-16T23:43:18.806 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.56987.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56987.log.gz 2024-09-16T23:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52410.log.gz 2024-09-16T23:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61920.log 2024-09-16T23:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36023.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46453.log 2024-09-16T23:43:18.808 INFO:teuthology.orchestra.run.smithi098.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36023.log.gz 2024-09-16T23:43:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.61920.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61920.log.gz 2024-09-16T23:43:18.810 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68676.log 2024-09-16T23:43:18.823 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50626.log 2024-09-16T23:43:18.824 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68676.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68676.log.gz 2024-09-16T23:43:18.838 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70899.log 2024-09-16T23:43:18.838 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50626.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50626.log.gz 2024-09-16T23:43:18.839 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.52957.log 2024-09-16T23:43:18.839 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.70899.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70899.log.gz 2024-09-16T23:43:18.852 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68549.log 2024-09-16T23:43:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.2.52957.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65923.log 2024-09-16T23:43:18.859 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68549.log: -- replaced with /var/log/ceph/cluster2-client.mirror.2.52957.log.gz 2024-09-16T23:43:18.861 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.68549.log.gz 2024-09-16T23:43:18.861 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40992.log 2024-09-16T23:43:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65923.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65923.log.gz 2024-09-16T23:43:18.862 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72626.log 2024-09-16T23:43:18.863 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.40992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72014.log 2024-09-16T23:43:18.864 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72626.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.72626.log.gz 2024-09-16T23:43:18.864 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73552.log 2024-09-16T23:43:18.865 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.72014.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.72014.log.gz 2024-09-16T23:43:18.866 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66797.log 2024-09-16T23:43:18.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.73552.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73552.log.gz 2024-09-16T23:43:18.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36698.log 2024-09-16T23:43:18.868 INFO:teuthology.orchestra.run.smithi098.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40992.log.gz 2024-09-16T23:43:18.868 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66797.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66797.log.gz 2024-09-16T23:43:18.868 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52562.log 2024-09-16T23:43:18.871 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36698.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36698.log.gz 2024-09-16T23:43:18.871 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64083.log 2024-09-16T23:43:18.871 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52562.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52562.log.gz 2024-09-16T23:43:18.871 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62828.log 2024-09-16T23:43:18.872 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.64083.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64083.log.gz 2024-09-16T23:43:18.872 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53516.log 2024-09-16T23:43:18.874 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.62828.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62828.log.gz 2024-09-16T23:43:18.874 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45863.log 2024-09-16T23:43:18.877 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53516.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53516.log.gz 2024-09-16T23:43:18.877 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68442.log 2024-09-16T23:43:18.894 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72203.log 2024-09-16T23:43:18.895 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.68442.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68442.log.gz 2024-09-16T23:43:18.908 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40877.log 2024-09-16T23:43:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72203.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72203.log.gz 2024-09-16T23:43:18.910 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41795.log 2024-09-16T23:43:18.916 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.40877.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.40877.log.gz 2024-09-16T23:43:18.916 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55959.log 2024-09-16T23:43:18.929 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.41795.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41795.log.gz 2024-09-16T23:43:18.932 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69154.log 2024-09-16T23:43:18.933 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.55959.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55959.log.gz 2024-09-16T23:43:18.936 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46704.log.gz 2024-09-16T23:43:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70526.log 2024-09-16T23:43:18.941 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69154.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69154.log.gz 2024-09-16T23:43:18.942 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50940.log 2024-09-16T23:43:18.942 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70526.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.70526.log.gz 2024-09-16T23:43:18.942 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62186.log 2024-09-16T23:43:18.953 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50940.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50940.log.gz 2024-09-16T23:43:18.954 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67247.log 2024-09-16T23:43:18.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.62186.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62186.log.gz 2024-09-16T23:43:18.958 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51313.log 2024-09-16T23:43:18.958 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67247.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67247.log.gz 2024-09-16T23:43:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50799.log 2024-09-16T23:43:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.51313.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51313.log.gz 2024-09-16T23:43:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46783.log 2024-09-16T23:43:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.50799.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50799.log.gz 2024-09-16T23:43:18.962 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45428.log 2024-09-16T23:43:18.970 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44494.log 2024-09-16T23:43:18.972 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45428.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48342.log.gz 2024-09-16T23:43:18.973 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45673.log.gz 2024-09-16T23:43:18.974 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36782.log 2024-09-16T23:43:18.979 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44494.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44494.log.gz 2024-09-16T23:43:18.982 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53355.log 2024-09-16T23:43:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.36782.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.36782.log.gz 2024-09-16T23:43:18.983 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47124.log 2024-09-16T23:43:18.984 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.53355.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53355.log.gz 2024-09-16T23:43:18.984 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.46997.log 2024-09-16T23:43:18.992 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47124.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47124.log.gz 2024-09-16T23:43:18.993 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52610.log 2024-09-16T23:43:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.1.46997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70125.log 2024-09-16T23:43:19.003 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52610.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52610.log.gz 2024-09-16T23:43:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49734.log 2024-09-16T23:43:19.023 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70125.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65365.log 2024-09-16T23:43:19.023 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.70125.log.gz 2024-09-16T23:43:19.024 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.49734.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49734.log.gz 2024-09-16T23:43:19.037 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52634.log 2024-09-16T23:43:19.050 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65365.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65365.log.gz 2024-09-16T23:43:19.055 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43433.log 2024-09-16T23:43:19.056 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52634.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52634.log.gz 2024-09-16T23:43:19.063 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53761.log 2024-09-16T23:43:19.073 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.43433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45487.log 2024-09-16T23:43:19.077 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.53761.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43433.log.gz 2024-09-16T23:43:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68276.log 2024-09-16T23:43:19.087 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45487.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45487.log.gz 2024-09-16T23:43:19.087 INFO:teuthology.orchestra.run.smithi098.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53761.log.gz 2024-09-16T23:43:19.093 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46942.log.gz 2024-09-16T23:43:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73402.log 2024-09-16T23:43:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.68276.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.68276.log.gz 2024-09-16T23:43:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66887.log 2024-09-16T23:43:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73402.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73402.log.gz 2024-09-16T23:43:19.102 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65737.log 2024-09-16T23:43:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66887.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.66887.log.gz 2024-09-16T23:43:19.103 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39428.log 2024-09-16T23:43:19.106 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65737.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65737.log.gz 2024-09-16T23:43:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61330.log 2024-09-16T23:43:19.111 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69195.log 2024-09-16T23:43:19.117 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61330.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.46997.log.gz 2024-09-16T23:43:19.117 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61330.log.gz 2024-09-16T23:43:19.118 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.39428.log.gz 2024-09-16T23:43:19.118 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60400.log 2024-09-16T23:43:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.69195.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.69195.log.gz 2024-09-16T23:43:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65116.log 2024-09-16T23:43:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66212.log 2024-09-16T23:43:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.65116.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60400.log.gz 2024-09-16T23:43:19.123 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70857.log 2024-09-16T23:43:19.123 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.65116.log.gz 2024-09-16T23:43:19.123 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.66212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39120.log 2024-09-16T23:43:19.123 INFO:teuthology.orchestra.run.smithi098.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66212.log.gz 2024-09-16T23:43:19.123 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69233.log 2024-09-16T23:43:19.136 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39120.log: /var/log/ceph/cluster1-client.mirror.70857.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.70857.log.gz 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39162.log 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.69233.log: 86.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69233.log.gz 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63681.log 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.39162.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55849.log 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.63681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52341.log 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55849.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55849.log.gz 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.70711.log 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63681.log.gz 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.52341.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47656.log 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52341.log.gz 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:43:19.137 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.70711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39254.log 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.70711.log.gz 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47656.log: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41320.log 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47656.log.gz 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.39254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61804.log 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.74500.log 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.61804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60103.log 2024-09-16T23:43:19.138 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.admin.74500.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.74500.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61804.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55911.log 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.60103.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60103.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72835.log 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.55911.log: 87.4% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39254.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55911.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39162.log.gz 2024-09-16T23:43:19.139 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67472.log 2024-09-16T23:43:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72835.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72835.log.gz 2024-09-16T23:43:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44193.log 2024-09-16T23:43:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.67472.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67472.log.gz 2024-09-16T23:43:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51717.log 2024-09-16T23:43:19.140 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.44193.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44193.log.gz 2024-09-16T23:43:19.142 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54028.log 2024-09-16T23:43:19.142 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.51717.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.51717.log.gz 2024-09-16T23:43:19.143 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47412.log 2024-09-16T23:43:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.54028.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54028.log.gz 2024-09-16T23:43:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45386.log 2024-09-16T23:43:19.145 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59595.log 2024-09-16T23:43:19.147 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.45386.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45386.log.gz 2024-09-16T23:43:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42919.log 2024-09-16T23:43:19.148 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.59595.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59595.log.gz 2024-09-16T23:43:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39001.log 2024-09-16T23:43:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.42919.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.42919.log.gz 2024-09-16T23:43:19.152 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41523.log 2024-09-16T23:43:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.admin.39001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72975.log 2024-09-16T23:43:19.155 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47450.log 2024-09-16T23:43:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.72975.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72975.log.gz 2024-09-16T23:43:19.156 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60175.log 2024-09-16T23:43:19.158 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.47450.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.39001.log.gz 2024-09-16T23:43:19.161 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48157.log.gz 2024-09-16T23:43:19.162 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47450.log.gz 2024-09-16T23:43:19.163 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63832.log 2024-09-16T23:43:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.60175.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60175.log.gz 2024-09-16T23:43:19.164 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36953.log.gz 2024-09-16T23:43:19.164 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48480.log 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41382.log 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.48480.log: /var/log/ceph/cluster2-client.mirror.63832.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48480.log.gz 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63832.log.gz 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.72343.log 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.41382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73381.log 2024-09-16T23:43:19.167 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46314.log 2024-09-16T23:43:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.73381.log: /var/log/ceph/cluster2-client.mirror.2.72343.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.72343.log.gz 2024-09-16T23:43:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50206.log 2024-09-16T23:43:19.171 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster2-client.mirror.46314.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41382.log.gz 2024-09-16T23:43:19.171 INFO:teuthology.orchestra.run.smithi098.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46314.log.gz 2024-09-16T23:43:19.171 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73381.log.gz 2024-09-16T23:43:19.172 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cluster1-client.mirror.50206.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50206.log.gz 2024-09-16T23:43:19.182 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45784.log.gz 2024-09-16T23:43:19.189 INFO:teuthology.orchestra.run.smithi098.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39120.log.gz 2024-09-16T23:43:19.190 INFO:teuthology.orchestra.run.smithi098.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.41320.log.gz 2024-09-16T23:43:19.214 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41523.log.gz 2024-09-16T23:43:19.227 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46453.log.gz 2024-09-16T23:43:19.278 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39363.log.gz 2024-09-16T23:43:19.359 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46783.log.gz 2024-09-16T23:43:19.392 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47412.log.gz 2024-09-16T23:43:19.477 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45863.log.gz 2024-09-16T23:43:19.527 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45428.log.gz 2024-09-16T23:43:20.187 INFO:teuthology.orchestra.run.smithi098.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-16T23:43:22.107 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41630.log.gz 2024-09-16T23:43:22.489 INFO:teuthology.orchestra.run.smithi096.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-16T23:43:23.229 INFO:teuthology.orchestra.run.smithi096.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-16T23:43:23.419 INFO:teuthology.orchestra.run.smithi096.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-16T23:43:23.421 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-16T23:43:23.421 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m8.275s 2024-09-16T23:43:23.421 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m24.143s 2024-09-16T23:43:23.421 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.917s 2024-09-16T23:43:25.078 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38476.log.gz 2024-09-16T23:43:25.221 INFO:teuthology.orchestra.run.smithi098.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39870.log.gz 2024-09-16T23:43:26.086 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40265.log.gz 2024-09-16T23:43:27.567 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40747.log.gz 2024-09-16T23:43:29.649 INFO:teuthology.orchestra.run.smithi098.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41240.log.gz 2024-09-16T23:45:21.831 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-16T23:45:23.005 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-16T23:45:40.041 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-16T23:45:40.043 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-09-16T23:45:40.043 INFO:teuthology.orchestra.run.smithi098.stderr:real 2m24.866s 2024-09-16T23:45:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:user 4m4.516s 2024-09-16T23:45:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m10.735s 2024-09-16T23:45:40.044 INFO:tasks.ceph:Archiving logs... 2024-09-16T23:45:40.044 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi096/log 2024-09-16T23:45:40.046 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T23:45:43.382 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi098/log 2024-09-16T23:45:43.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T23:46:19.729 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T23:46:19.742 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T23:46:19.742 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T23:46:19.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T23:46:19.927 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T23:46:19.941 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T23:46:19.941 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T23:46:19.944 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T23:46:19.959 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-16T23:46:19.962 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-09-16T23:46:19.963 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T23:46:19.963 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-16T23:46:19.964 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 7 377 2 +3384us[+3385us] +/- 38ms 2024-09-16T23:46:19.964 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 4 -4036us[-4035us] +/- 65ms 2024-09-16T23:46:19.964 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -2388us[-2388us] +/- 85ms 2024-09-16T23:46:19.964 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:46:19.987 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-16T23:46:19.988 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-09-16T23:46:19.988 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +3362us[+3364us] +/- 38ms 2024-09-16T23:46:19.988 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -4006us[-4003us] +/- 65ms 2024-09-16T23:46:19.988 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -2379us[-2377us] +/- 85ms 2024-09-16T23:46:19.988 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-16T23:46:19.989 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T23:46:20.001 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T23:46:20.001 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T23:46:20.043 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T23:46:20.112 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-16T23:46:20.216 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T23:46:20.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T23:46:20.247 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-16T23:46:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T23:46:20.276 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T23:46:20.292 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T23:46:20.304 INFO:teuthology.task.internal:Duration was 2820.084587 seconds 2024-09-16T23:46:20.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T23:46:20.316 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T23:46:20.316 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T23:46:20.319 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T23:46:20.359 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T23:46:20.394 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-16T23:46:20.777 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T23:46:20.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-09-16T23:46:20.778 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T23:46:20.841 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-09-16T23:46:20.841 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T23:46:20.862 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T23:46:20.862 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T23:46:20.884 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T23:46:20.930 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T23:46:20.930 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T23:46:20.953 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T23:46:21.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T23:46:21.104 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T23:46:21.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T23:46:21.129 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T23:46:21.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T23:46:21.184 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T23:46:21.186 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T23:46:21.207 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-09-16T23:46:21.250 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-09-16T23:46:21.264 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T23:46:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T23:46:21.295 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T23:46:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T23:46:21.318 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T23:46:21.331 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T23:46:21.332 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi096 2024-09-16T23:46:21.332 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T23:46:21.368 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2024-09-16_19:45:14-rbd-wip-dis-testing-distro-default-smithi/7907656/remote/smithi098 2024-09-16T23:46:21.369 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T23:46:21.402 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T23:46:21.402 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T23:46:21.405 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T23:46:21.453 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T23:46:21.467 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T23:46:21.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T23:46:21.481 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T23:46:21.482 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T23:46:21.484 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T23:46:21.497 INFO:teuthology.orchestra.run.smithi096.stdout: 265861 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 23:46 /home/ubuntu/cephtest 2024-09-16T23:46:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 23:46 /home/ubuntu/cephtest 2024-09-16T23:46:21.510 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T23:46:21.522 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T23:46:21.610 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 2820.084587097168 flavor: default owner: scheduled_dis@teuthology success: true 2024-09-16T23:46:21.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T23:46:21.688 INFO:teuthology.run:pass